Methods and Systems for Performing Image Analytics Using Graphical Reporting Associated with Clinical Images

ABSTRACT

Methods and systems for performing image analytics using graphical reporting associated with clinical images. One system includes at least one data source and a server. The server includes an electronic processor and an interface for communicating with the data source. The electronic processor is configured to receive training information from the at least one data source over the interface. The training information includes a plurality of images and graphical reporting associated with each of the plurality of images. Each graphical reporting includes a graphical marker designating a portion of one of the plurality of images and diagnostic information associated with the portion of the one of the plurality of images. The electronic processor is also configured to perform machine learning to develop a model using the training information. The model is used to automatically analyze an image.

RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application Nos.62/174,978; 62/174,962; 62/174,956; 62/174,953; and 62/174,946 all filedJun. 12, 2015, and the entire content of each priority application isincorporated by reference herein.

FIELD

Embodiments of the present invention relate to systems and methods forperforming image analytics using machine learning.

BACKGROUND

The number of medical images obtained worldwide continues to rapidlygrow due to enhancements in imaging technology, including magneticresonance imaging (“MRI”), spiral computed tomography (“CT”), positionemission tomography (“PT”), ultrasound, various forms of radiography,mammography, breast tomosynthesis, medical photography and mobileimaging applications. The number of images generated using these andother technologies surpass the capacity of expert physicians (e.g.,radiologists) to individually analyze each image. For example, a CT scanof an abdomen that once included 100 or fewer images now commonlyincludes 1,000 to 3,000 images. A physician who reports 50 CTs, MRIs, orPT scans in a day may now be required to view and analyze approximately100,000 medical images from the current exams in addition to relevantprior exams for comparison. Also, there is now more clinical data frommore sources that must also be integrated with imaging information.

Rather than simply replicating and speeding existing human processes,computers may simultaneously process multiple tasks and draw uponmultiple simultaneous information sources based on interactive rules.Therefore, unlike the human brain, which is largely a serial processor,multi-tasking computer system may simultaneously weigh many factors, andtherefore complement or exceed human performance with regard to medicalimage interpretation.

The ever-growing need to increase the speed and accuracy of medicalimage review, including the integration and consideration of clinicaland reference data, demands improvements to medical image andinformation management systems. Methods and technologies describedherein are therefore necessary and welcome additions to the art.

SUMMARY

Embodiments of the present invention provide systems and methods fordeveloping image analytics using machine learning. For example, someembodiments of the invention use graphical reporting to train a learningengine. Graphical reporting includes images combined with structureddata that allows the learning engine to more rapidly “learn” imagecharacteristics and associated diagnoses. The learning engine may beexecuted by a computer system and may be configured to obtain traininginformation, including graphical reporting, from one or more datasources. The sources may be connected to the computer system over one ormore network connections, such as the Internet or other public orprivate networks.

As the learning engine is progressively trained, the learning engine(i.e., the models developed by the learning engine) may be used toanalyze medical images and associated data to provide diagnosticinformation. In some embodiments, the diagnostic information may includecategorization of an image or exam (comprised of one or more images)into various categories, such as: (1) normal, (2) abnormal, and (3)indeterminate. The models developed by the learning engine may be usedin various medical scenarios, including emergency rooms, urgent carecenters, clinics, nursing care facilities, home care locations, ruralareas, third-world countries, and any other environment where medicalimages are performed without the benefit of an immediately availableprofessional specializing in medical imaging. The models may be used tosupplement a professional, replace a professional, assist aprofessional, triage images or exams for a professional,aggregate/analyze a professional's performance, and combinationsthereof. For example, the models may be used to identify images in anexam considered are normal and, therefore, do not require professionalinterpretation, exams considered abnormal, exams requiring emergentattention, image regions for a particular image type that are mostlikely to show abnormalities, and combinations thereof withprobabilities influenced via simultaneous aggregation and considerationof clinical, demographic, and external data (e.g., local epidemics).

Accordingly, embodiments of the invention combine graphical reportingwith clinical data and deep machine learning to automatically createmodels that simultaneously weigh many factors to automatically analyzeclinical images. The models may be used to triage medical imaging examsor images within an exam to categorize and indicate those images orexams that do not require human review, those images that require addedhuman attention, those images that require routing to experts, or thoseimages that may be useful as a reference image (e.g., an image used forteaching, marketing, patient education, or public health) that should berouted to one or more repositories. The models may also be used toautomatically identify the most critical images within an imaging exam,most critical regions of images, or both based on clinical indicationsand pre-test probabilities that are derived from demographic, clinical,and external data. The models may also be used to generate automatedpre-test and post-test probability reports relative to relevantdiagnostic questions or configurable specified questions (e.g., “Isthere a tumor?,” “Is there a fracture?,” “Is there a tube malposition?,”and the like). The models may also parse images and exams intocategories, such as normal, abnormal, and indeterminate, and mayautomatically select and present best comparison images or exams orindicate the most relevant image regions for comparison. The models maymake these selections based on an analysis of data includingindications, demographics, risks, clinical data, epidemiological data,or a combination thereof. Similarly, in some embodiments, the models areused to automatically present the best image plane for volumetric imagesor a comparison of volumetric images or to automate comparing exams,images, and image regions to best detect changes over time (e.g., newemerging cancer, aneurysm, etc.) or draw inferences about the contentsof a tissue or lesion (e.g., this mass is probably composed of fat, thismass is probably benign, etc.). Output from the models may be used toautomatically notify users of relevant clinical events that may impactthe interpretation of image changes (e.g., this lesion is smaller butthere was a de-bulking surgery since the last exam so it is not clearwhether the chemotherapy is working).

In some embodiments, the models may also be applied based on rules andother configurable settings linked to a user (e.g., a diagnosingphysician), a facility, an organization, an exam type, a body part, ageographic location, a season, and the like. Embodiments may alsoprovide standards-based interoperable reporting of results and imagelabeling performed by the models to facilitate a close-loop feedbackmechanism from users (e.g., PACS users). Embodiments may also provideautomated outcome analysis and performance metrics, which may be judgedby models developed using the learning engine.

For example, one embodiment provides a system for performing imageanalytics using graphical reporting associated with clinical images. Thesystem includes at least one data source and a server. The serverincludes an electronic processor and an interface for communicating withthe data source. The electronic processor is configured to receivetraining information from the at least one data source over theinterface. The training information includes a plurality of images andgraphical reporting associated with each of the plurality of images.Each graphical reporting includes a graphical marker designating aportion of one of the plurality of images and diagnostic informationassociated with the portion of the one of the plurality of images. Theelectronic processor is also configured to perform machine learning todevelop a model using the training information. The model is used toautomatically analyze an image.

Other aspects of the invention will become apparent by consideration ofthe detailed description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system for performing image analytics using machinelearning according to some embodiments.

FIG. 2 is a flowchart of a method of performing image analytics usinggraphical reporting associated with clinical images according to someembodiment.

FIGS. 3 and 4 are images including a graphical marker according to someembodiments.

FIG. 5 is a flowchart of a method of automatically analyzing clinicalimages using image analytics developed using graphical reportingassociated with previously-analyzed clinical images according to someembodiments.

FIG. 6 is a flowchart of a method of automatically analyzing clinicalimages and determining when additional imaging may aid a diagnosisaccording to some embodiments.

FIG. 7 is a flowchart of a method of automatically determining clinicalimages within an image study for display to a diagnosing physicianaccording to some embodiments.

FIG. 8 is a flowchart of a method of automatically determining portionsof clinical images for display to a diagnosing physician according tosome embodiments.

FIG. 9 is a flowchart of a method of automatically analyzing clinicalimages using rules and image analytics developed using graphicalreporting associated with previously-analyzed clinical images accordingto some embodiments.

FIG. 10 is a flowchart of a method of automatically scoring diagnosesassociated with clinical images according to some embodiments.

FIG. 11 is a flowchart of a method of automatically determiningdiagnosis discrepancies for clinical images according to someembodiments.

FIG. 12 is a flowchart of a method of automatically determining apotential bias of a diagnosing physician according to some embodiments.

FIG. 13 is a flowchart of a method of automatically determining imagecharacteristics serving as a basis for a diagnosis associated with animage study type according to some embodiments.

FIG. 14 is a flowchart of a method of automatically selecting an implantfor a patient planning to undergo a procedure involving placement of theimplant according to some embodiments.

FIG. 15 is a flowchart of a method of mapping pathology results toclinical images according to some embodiments.

FIG. 16 illustrates an image including a plurality of marked biopsylocations according to some embodiments.

FIG. 17 illustrates a pathology report according to some embodiments.

FIGS. 18 and 19 illustrate graphical user interfaces displayingstatistics of clinical outcomes according to some embodiments.

FIG. 20 illustrates a graphical user interface displaying a tablelinking imaging results with pathology results according to someembodiments.

FIG. 21 is a flowchart of a method of mapping one or more biopsylocations to pathology results according to some embodiments.

Other aspects of the invention will become apparent by consideration ofthe detailed description.

DETAILED DESCRIPTION

Before embodiments of the invention are explained in detail, it is to beunderstood that the invention is not limited in its application to thedetails of construction and the arrangement of components set forth inthe following description or illustrated in the accompanying drawings.The invention is capable of other embodiments and of being practiced orof being carried out in various ways.

Also, it is to be understood that the phraseology and terminology usedherein is for the purpose of description and should not be regarded aslimiting. The use of “including,” “comprising” or “having” andvariations thereof herein is meant to encompass the items listedthereafter and equivalents thereof as well as additional items. Theterms “mounted,” “connected” and “coupled” are used broadly andencompass both direct and indirect mounting, connecting and coupling.Further, “connected” and “coupled” are not restricted to physical ormechanical connections or couplings, and may include electricalconnections or couplings, whether direct or indirect. Also, electroniccommunications and notifications may be performed using any known meansincluding direct connections, wireless connections, etc.

A plurality of hardware and software based devices, as well as aplurality of different structural components may be utilized toimplement the invention. In addition, embodiments of the invention mayinclude hardware, software, and electronic components or modules that,for purposes of discussion, may be illustrated and described as if themajority of the components were implemented solely in hardware. However,one of ordinary skill in the art, and based on a reading of thisdetailed description, would recognize that, in at least one embodiment,the electronic-based aspects of the invention may be implemented insoftware (e.g., stored on non-transitory computer-readable medium)executable by one or more processors. As such, it should be noted that aplurality of hardware and software based devices, as well as a pluralityof different structural components, may be utilized to implement theinvention. For example, “mobile device,” “computing device,” and“server” as described in the specification may include one or moreelectronic processors, one or more memory modules includingnon-transitory computer-readable medium, one or more input/outputinterfaces, and various connections (e.g., a system bus) connecting thecomponents.

Machine learning generally refers to the ability of a computer programto learn without being explicitly programmed. In some embodiments, acomputer program (e.g., a learning engine) is configured to construct amodel (e.g., one or more algorithms) based on example inputs. Supervisedlearning involves presenting a computer program with example inputs andtheir desired (e.g., actual) outputs. The computer program is configuredto learn a general rule (e.g., a model) that maps the inputs to theoutputs. The computer program may be configured to perform machinelearning using various types of methods and mechanisms. For example, thecomputer program may perform machine learning using decision treelearning, association rule learning, artificial neural networks,inductive logic programming, support vector machines, clustering,Bayesian networks, reinforcement learning, representation learning,similarity and metric learning, sparse dictionary learning, and geneticalgorithms. Using all of these approaches, a computer program mayingest, parse, and understand data and progressively refine models fordata analytics.

Providing a learning engine with the proper example inputs and outputsmay be a challenge, and analytics provided by a learning engine aregenerally only as good as the example inputs and outputs (hereinafterreferred to as the “training information”) provided. For example, whenteaching a learning engine to analyze images by providing the learningengine with training information that includes previously-analyzedimages, the learning engine needs to be able to identify what portionsof an image are relevant for a particular diagnosis and what portionsare irrelevant. The learning engine may benefit from knowing whether animage is associated with an abnormal diagnosis or a normal diagnosis andwhat clinical, demographic, or external data may be used to analyze thepre-test probability of a particular disease or finding. Historically,this information was not readily available for a particular medicalimage in ways that allowed a learning engine to effectively andefficiently learn from images.

Accordingly, embodiments of the invention provide methods and systemsfor providing training information to a learning engine to properlytrain the learning engine to automatically analyze medical images andprovide diagnosis support. In particular, embodiments of the inventionprovide improvements to existing machine learning image processingtechnology by providing a learning engine with training information thatinforms the learning engine what portions of an image to analyze andinformation regarding findings (e.g., the contained normal/abnormalfinding), diagnoses (e.g., a diagnosis or a differential diagnosis),confidence or probability rating of observations, and combinationsthereof. Similarly, the training information provided to a learningengine may specify relevant images or portions of images from comparisonimaging exams. The learning engine may use this information to detectdiagnostic changes over time and to learn to compare images to derivediagnostic information. Thus, the learning engine may learn to detect anemerging breast cancer by comparing serial mammograms on the samepatient or by comparing a left breast image to a corresponding rightbreast image taken from the same projection on the same day. Similarly,the learning engine may learn that a lesion yielding high signal on aT1-weighted MRI and low signal on a fast suppressed MRI taken at aboutthe same time is likely composed of lipid. While learning, the learningengine may also simultaneously consider clinical and demographic datathat comprise risk factors influencing the probability of a diagnosis orfinding.

FIG. 1 illustrates a system 100 for performing machine learningaccording to some embodiments of the invention. The system 100 includesa server 102 that includes a plurality of electrical and electroniccomponents that provide power, operational control, and protection ofthe components within the server 102. For example, as illustrated inFIG. 1, the server 102 may include an electronic processor 104 (e.g., amicroprocessor, application-specific integrated circuit (ASIC), oranother suitable electronic device), a memory 106 (e.g., anon-transitory, computer-readable storage medium), and an input/outputinterface 108. The electronic processor 104, the memory 106, and theinput/output interface 108 communicate over one or more connections orbuses. The server 102 illustrated in FIG. 1 represents one example of aserver and embodiments described herein may include a server withadditional, fewer, or different components than the server 102illustrated in FIG. 1. Also, in some embodiments, the server 102performs functionality in addition to the functionality describedherein. Similarly, the functionality performed by the server 102 (i.e.,through execution of instructions by the electronic processor 104) maybe distributed among multiple servers. Accordingly, functionalitydescribed herein as being performed by the electronic processor 104 maybe performed by one or more electronic processors included in the server102, external to the server 102, or a combination thereof.

The memory 106 may include read-only memory (“ROM”), random accessmemory (“RAM”) (e.g., dynamic RAM (“DRAM”), synchronous DRAM (“SDRAM”),and the like), electrically erasable programmable read-only memory(“EEPROM”), flash memory, a hard disk, a secure digital (“SD”) card,other suitable memory devices, or a combination thereof. The electronicprocessor 104 executes computer-readable instructions (“software”)stored in the memory 106. The software may include firmware, one or moreapplications, program data, filters, rules, one or more program modules,and other executable instructions. For example, the software may includeinstructions and associated data for performing the methods describedherein. For example, as illustrated in FIG. 1, the memory 106 may storea learning engine 110 (i.e., software) for performing image analytics asdescribed herein (e.g., processing training information to developmodels). However, in other embodiments, the functionality describedherein as being performed by the learning engine 110 may be performedthrough one or more software modules stored in the memory 106 orexternal memory.

The input/output interface 108 allows the server 102 to communicate withdevices external to the server 102. For example, as illustrated in FIG.1, the server 102 may communicate with one or more data sources 112through the input/output interface 108. In particular, the input/outputinterface 108 may include a port for receiving a wired connection to anexternal device (e.g., a universal serial bus (“USB”) cable and thelike), a transceiver for establishing a wireless connection to anexternal device (e.g., over one or more communication networks 111, suchas the Internet, a local area network (“LAN”), a wide area network(“WAN”), and the like), or a combination thereof.

In some embodiments, the server 102 also receives input from one or moreperipheral devices, such as a keyboard, a pointing device (e.g., amouse), buttons on a touch screen, a scroll ball, mechanical buttons,and the like through the input/output interface 108. Similarly, in someembodiments, the server 102 provides output to one or more peripheraldevices, such as a display device (e.g., a liquid crystal display(“LCD”), a touch screen, and the like), a printer, a speaker, and thelike through the input/output interface 108. In some embodiments, outputmay be provided within a graphical user interface (“GUI”) (e.g.,generated by the electronic processor 104 executing instructions anddata stored in the memory 106 and presented on a touch screen or otherdisplay) that enables a user to interact with the server 102. In otherembodiments, a user may interact with the server 102 through one or moreintermediary devices, such as a personal computing device laptop,desktop, tablet, smart phone, smart watch or other wearable, smarttelevision, and the like). For example, a user may configurefunctionality performed by the server 102 as described herein byproviding data to an intermediary device that communicates with theserver 102. In particular, a user may use a browser application executedby an intermediary device to access a web page that receives input fromand provides output to the user for configuring the functionalityperformed by the server 102.

As illustrated in FIG. 1, the system 100 includes one or more datasources 112. Each data source 112 may include a plurality of electricaland electronic components that provide power, operational control, andprotection of the components within the data source 112. In someembodiments, each data source 112 represents a server, a database, apersonal computing device, or a combination thereof. For example, asillustrated in FIG. 1, each data source 112 may include an electronicprocessor 113 (e.g., a microprocessor, ASIC, or other suitableelectronic device), a memory 114 (e.g., a non-transitory,computer-readable storage medium), and an input/output interface 116.The data sources 112 illustrated in FIG. 1 represents one example ofdata sources and embodiments described herein may include a data sourcewith additional, fewer, or different components than the data sources112 illustrated in FIG. 1. Also, in some embodiments, the server 102communicates with more or fewer data sources 112 than illustrated inFIG. 1.

The input/output interface 116 allows the data source 112 to communicatewith external devices, such as the server 102. For example, asillustrated in FIG. 1, the input/output interface 116 may include atransceiver for establishing a wireless connection to the server 102 orother devices through the communication network 111 described above.Alternatively or in addition, the input/output interface 116 may includea port for receiving a wired connection to the server 102 or otherdevices. Furthermore, in some embodiments, the data sources 112 alsocommunicate with one or more peripheral devices through the input/outputinterface 116 for receiving input from a user, providing output to auser, or a combination thereof. In other embodiments, one or more of thedata sources 112 may communicate with the server 102 through one or moreintermediary devices. Also, in some embodiments, one or more of the datasources 112 may be included in the server 102.

The memory 114 of each data source 112 may store medical data, such asmedical images (i.e., clinical images) and associated data (e.g.,reports, metadata, and the like). For example, the data sources 112 mayinclude a picture archiving and communication system (“PACS”), aradiology information system (“RIS”), an electronic medical record(“EMR”), a hospital information system (“HIS”), an image study orderingsystem, and the like. In some embodiments, as noted above, data storedin the data sources 112 or a portion thereof may be stored locally onthe server 102 (e.g., in the memory 106).

FIG. 2 is a flowchart illustrating a method 200 performed by the server102 (i.e., the electronic processor 104 executing instructions, such asthe learning engine 110) for automatically performing image analyticsusing graphical reporting associated with clinical images according tosome embodiments. As described below, the learning engine 110 performsdata analytics to discover meaningful patterns in images and buildsmodels based on these discovered patterns, which can be used toautomatically analyze images or other medical data. As illustrated inFIG. 2, the method 200 includes receiving, at the learning engine 110,training information (at block 202). The training information may takevarious forms, and in some embodiments, the training informationincludes data stored in the data sources 112. For example, the traininginformation may include one or more images. The images may includeradiographic images, magnetic resonance (“MR”) images (“MRI”),ultrasonography (“US”) images, endoscopy images, elastography images,tactile images, thermography, medical photography (e.g., photographs ofskin conditions and other surface conditions, such as cleft palates,birth marks, moles, dislocations, and the like), computed tomography(“CT”) images, electrocardiography (“ECG”) data, position emissiontomography (“PT”) images, and the like. In some embodiments, the imagesprovided to the learning engine 110 as training information may bepre-processed to improve training. For example, 2-D axial images may bepreprocessed using a maximum intensity projection (“MIP”) ormulti-planar reconstruction (“MPR”) algorithm. Also, in someembodiments, the learning engine 110 may be trained using multiple viewsof the same anatomy (e.g., axial, sagittal, coronal, or oblique planes).Also, the learning engine 110 may be trained using 3-D, 4-D, or even 5-Ddatasets, which are commonly collected as part of an imaging procedurebut seldom used by diagnosing physicians.

In some embodiments, the training information also includes graphicalreporting. Graphical reporting may refer to a method of reporting wherean area of an image is graphically marked and the area is associatedwith diagnostic information (e.g., indicating that the identified areaof the image illustrates a particular abnormality or a particularnormality). The diagnostic information associated with the graphicalmarker may be structured or unstructured and may be in the form of text(generated manually by a diagnosing physician, automatically by acomputer system, or a combination thereof), audio, video, images, andthe like. The graphical marker may be created manually by a diagnosingphysician, such as a radiologist, a cardiologist, a physician'sassistant, a technologist, and the like or automatically by a computersystem, such as a PACS. Similarly, the associated diagnostic informationmay be created manually by a diagnosing physician, such as aradiologist, a cardiologist, a physician's assistant, a technologist,and the like or automatically by a computer system, such as a PACS.

For example, a diagnosing physician reading a mammogram may mark one ormore abnormalities on one or more of the images generated from themammogram (a typical breast tomosynthesis exam contains approximately300 or fewer images), such as by circling calcifications in one or moreof the images. The diagnosing physician may also associate diagnosticinformation with each circled calcifications, such as the classificationof “suspicious pleomorphic calcifications.” The diagnosing physician mayalso generate a report that includes structured information describingimage findings based on a particular lexicon, such as the AmericanCollege of Radiology's (“ACR's”) BI-RADS lexicon.

FIG. 3 illustrates an example graphical marker 300 added to an image302. As illustrated in FIG. 3, the graphical marker 300 includes acircle surrounding an area of the image 302 of interest to thediagnosing physician. In addition to creating the graphical marker 300,a diagnosing physician (or a computer) provides diagnostic informationassociated with the area of the image 302 represented by the graphicalmarker 300. As noted above, the diagnostic information may be dictatedby a diagnosing physician, selected by a diagnosing physician from adrop-down menu, generated by a PACS, or a combination thereof. Althoughthe graphical marker 300 is illustrated in FIG. 3 as a circle, thegraphical marker 300 may take different shapes and sizes, such as apoint, an arrow, a line, a sphere, a rectangle, a cone, and the like andmay be a one-dimensional marker, a two-dimensional marker, or athree-dimensional marker. For example, in some embodiments, thegraphical marker 300 may follow a border or highlight an objectrepresented in the image 302. For example, as illustrated in FIG. 4, thegraphical marker 300 follows the border of the brainstem pons. In someembodiments, the shape or size of the graphical marker 300 may bemanually set by the user performing the graphical reporting. In otherembodiments, the shape or size of the graphical marker 300 may beautomatically set by the graphical reporting software, such as based onthe anatomy represented in the image 302, a portion of the imageselected by the user, and the like. Also, an image may include one ormore multiple graphical markers.

The images and the associated graphical reporting may be provided to thelearning engine 110 to allow the learning engine 110 to rapidly learn todistinguish between a normal image and an abnormal image. In particular,by providing the learning engine 110 with the images and the associatedgraphical reporting, the learning engine 110 may associate a particularportion of an image with a particular diagnosis by knowing where to lookwithin a particular image and how the marked area of the image wasdiagnosed. In other words, the graphical reporting provides the “where”(through the graphical marker) and the associated “what” (through thediagnostic information associated with the graphical marker), whichallows the learning engine 110 to rapidly learn to analyze images. Inparticular, machine learning techniques that use images andcorresponding structured reports to perform image analytics are missingan association between a diagnosis included in a structure report andareas of the image. For example, the structured report may indicate thatthe image is associated with a particular diagnosis (e.g., normal orabnormal) but does not specify an area of the image that led to thediagnosis.

As noted above, each graphical marker 300 may be associated withdiagnostic information. In some embodiments, the diagnostic informationis a classification or a delineation and classification over a spectrum(e.g., from benign to cancerous) that optionally includes an associatedprobability. For example, the diagnostic information may include aclassification of a plurality of categories, such as a normal category,an abnormal category, or an indeterminate category. The classificationmay be based on the associated images, such as lesion morphology (e.g.,a mass, a non-mass, or a focus lesion), lesion shape (e.g., round, oval,irregular, circumscribed, or spiculated), image properties of a tissue(e.g., homogeneous or heterogeneous distribution of image enhancement),or combinations thereof. In some cases, the classification may definedynamic imaging properties of a contrast agent (e.g., rate of wash-inand wash-out of contrast agent or the rate of contrast change on theborders of the lesion). In other cases, the classification may be basedon the dynamic or static uptake of a radiopharmaceutical. Theclassification may also include the location of an area of interest withrespect to an anatomical coordinate system, such as the upper-leftquadrant of an organ. In some embodiments, the classification mayspecify an orientation of an area of interest, such as its alignmentwith anatomical structures (e.g., ducts). The classification may alsospecify a texture. In addition, in some embodiments, the classificationindicates the results of blood tests, pathology results, genetic tests,or other clinical scores.

In addition to or as an alternative to a classification, the diagnosticinformation associated with a graphical marker may indicate a diagnosis,confidence level, a probability, a histology, a tumor grade, a stage, adifferential diagnosis, a type of finding, an anatomical structure(e.g., organ(s) affected or other anatomical structures), an associatedclinical syndrome, associated demographic information (such as age,gender, race/ethnicity, etc.), family history, problem lists,medications, allergies, immunizations, past therapies, and combinationsthereof of that may be manually input or automatically pulled from apatient's electronic medical record. The diagnostic information may alsoinclude a measurement or a location within an image. The diagnosticinformation may also include morphological characteristics of an anomalydetected within an image.

In some embodiments, the training information also includes metadataassociated with images. The metadata may include header information,such as a Digital Imaging and Communications in Medicine (“DICOM”)header file, associated with an image included in the traininginformation. The metadata may specify the “how” for the image, such asthe imaging modality used, whether a contrast agent was used, theresolution or other settings for the imaging procedure, the imagingplane, the imaging technique, the field-of-view, slice thickness, date,and other imaging procedure or examination data. For example, a DICOMheader file may specify what part of anatomy was being imaged. In someembodiments, when the metadata received by the learning engine 110 doesnot include anatomy information, the learning engine 110 may beconfigured to process the image and automatically identify the anatomyincluded in the image. Accordingly, in some embodiments, the learningengine 110 may be configured to generate metadata for received images.

The training information may also include imaging procedure orexamination information associated with an image included in thetraining information. In some embodiments, the learning engine 110 mayreceive imaging procedure or examination data from an ordering systemwhere the order for the image study originated. In some embodiments, theimaging procedure or examination information also includes informationabout who performed the imaging procedure (e.g., imaging clinicinformation, imaging technician information, and the like), or whoanalyzed the results of the imaging procedure (e.g., who generated thegraphical reporting, such as an identifier of the diagnosing physicianor an indication that the graphical reporting was automaticallygenerated).

The training information may also include an imaging property of animage included in the training information. The imaging property mayinclude an indication of whether a contrast agent or a radioactiveisotope was used, a time after an agent or isotope was introduced, anorientation, and an image acquisition parameter (e.g., a MRI or CTradiation level).

The training information may also include patient information. In someembodiments, the patient information may be included in metadataassociated with the images, such as the DICOM header file. In otherembodiments, the patient information may be included separate from theimages, such as in an HIS, EMR system, and the like. The patientinformation may specify the “who” for the image and may specify patientdemographic information (e.g., age, gender, ethnicity, currentresidence, past residence, medical history, family medical history,biometrics, and the like). In some embodiments, the patient informationmay also include publicly-available patient information, such asinformation provided by the U.S. Center for Disease Control, WorldHealth Organization, or other public health organizations, such asepidemics, trends, and the like. The patient information may also beobtained from non-medical sources, such as information available throughsocial networking environments or other public information or datasources.

The training information may also include one or more reports (e.g.,structured or unstructured) associated with the images. These associatedreports may include an order, a DICOM structured radiology report, apathology report, a test result (e.g., blood test results), and thelike. For example, an MRI may be obtained after a potential cancerouslesion is detected on a medical image (e.g., an x-ray). The traininginformation may include the image where the cancerous lesion wasdetected as well as the subsequent MRI findings. Similarly, a biopsypathology result of the lesion may be included in the traininginformation if available. As described below in more detail, thelearning engine 110 may use the information about the lesion from bothimages and the pathology results when available to determine theaccuracy of the image-based diagnosis. Accordingly, using this metadataallows the learning machine 110 to better perform multi-factorialdiagnoses than a human diagnosing physician.

The reports may also include order records for the image study. Forexample, the order placed for a particular image study often providesinformation regarding what piece of anatomy was captured in the imagesand the reason for the image. If the reports are in text form, thelearning engine 110 may use natural language processing or otherdocument analysis technology to obtain metadata for a particular image.In some embodiments, the associated reports may include image studiesfor the patient associated with an image provided in the traininginformation. For example, information relating to a patient'sdemographic, race, history, risk factors, family history, problem list,smoking history, allergies, lab test results, or prior imaging resultsmay be used as training information for the learning engine 110.

The learning engine 110 may be configured to receive the traininginformation from one or more data sources 112, including, for example, aPACS, a vendor neutral archive, a RIS, an EMR, a HIS, an image studyordering system, a computer assisted detection (“CAD”) system, or acombination thereof. For example, in some embodiments, the learningengine 110 may receive the images and the associated graphical reportingfrom a PACS, RIS, and the like. The learning engine 110, however, mayreceive the other training information from devices different from thedevices providing the images and associated graphical reporting. Forexample, when the learning engine 110 receives an image, the learningengine 110 may query one or more devices or systems, such as a HIS, anEMR, and the like, for imaging procedure or examination information orpatient information. Also, in some embodiments, when performing thegraphical reporting, the computer system may prompt the diagnosingphysician for training information, such as a diagnosis, a measurement,an anatomy identification, patient information, imaging procedure orexamination information, and the like.

In some embodiments, the systems generating the images for the learningengine 110 may be configured to collect training information andtransmit the training information to the learning engine 110. In otherembodiments, an intermediary device may be configured to receive animage from an image source (e.g., a PACS), access one or more sources toobtain additional training information, and generate a package ofinformation for the learning engine 110. For example, in someembodiments, when an order is placed for an image study, a softwareapplication may be notified of the order, and the software applicationmay store the order or information obtained from the order and wait forthe corresponding images. For example, in some embodiments, an orderoften lists a destination for the resulting image study and associatedreport (e.g., the ordering physician, a specialist, a colleague, or thelike). Therefore, when an order is (e.g., electronically) placed andsubmitted (e.g., manually or automatically), the order information maybe automatically updated to include the learning engine 110 as adesignated destination. Therefore, when an image report is created andassociated with the image study (e.g., including a link to theassociated image study), the report may be automatically transmitted tothe learning engine 110 as training information. For example, in someembodiments the learning engine 110 may be configured to extract textfrom the image report and use a link included in the image report toaccess the associated image study.

Returning to the method 200 illustrated in FIG. 2, after the learningengine 110 receives the training information (at block 202), thelearning engine 110 performs machine learning to develop a model usingthe received training information (at block 204). For example, in someembodiments, the learning engine 110 identifies an area of each imageincluded in the training information and the associated diagnosticinformation based on the graphical reporting. For example, the graphicalreporting may be represented as a data structure that includescoordinates of the graphical marker 300 and the associated diagnosticinformation (e.g., a categorization, a measurement, a probability, arisk factor, and the like). The data structure may also store anidentifier of the image to which the data structure applies and also anidentifier of the overall image study to which the data structureapplies. Accordingly, the data records may be provided to the learningengine 110 separate from the associated images, and the learning engine110 may be configured to apply the data structure to the image toidentify the area of the image marked by the diagnosing physician andthe corresponding diagnostic information. Alternatively or in addition,the graphical marker or indicator provided by a diagnosing physician maybe superimposed on the image provided to the learning engine 110.Accordingly, the learning engine 110 may process the image to identifythe marked region, such as by processing pixels to detect a region ofpixels matching a particular shape, brightness, or color, or may employmeta-data to identify the coordinates of the marked region. In theseconfigurations, images without the associated graphical marker orindicator may also be provided to the learning engine 110 (e.g., toallow the learning engine 110 to process the images without anyinterference that may be caused by the graphical marker placed on theimages).

The learning engine 110 may also generate a data structure thatassociates an image with the graphical reporting (i.e., an area of theimage and associated diagnostic information). The data structure mayalso include the additional training information received by thelearning engine 110, such as the metadata, imaging procedure orexamination information, patient information, or a combination thereof.The learning engine 110 may use some of the training information todevelop a model for particular types of images or image studies. Forexample, when the DICOM header file indicates that an image is a CTspinal image, the learning engine 110 may group this traininginformation with other training information for CT spinal images, whichthe learning engine 110 may use to develop a customized model associatedwith CT spinal images. Similarly, the learning engine 110 may beconfigured to develop different models for different types of images(e.g., ECG data versus CT images) regardless of the underlying anatomy.In addition, the learning engine 110 may use patient demographicinformation to group training information associated with abdomen scansfor 20-year-old women separately from training information associatedwith abdomen scans for 60-year-old men. Also, in some embodiments, thelearning engine 110 may develop models that consider external factors,such as geographic location, season, existing epidemics, or communityimmunization patterns.

Including graphical reporting in the training information may alsoenable the learning engine 110 to learn to inspect a particular portionof a comparison image. For example, a diagnosing physician mayexplicitly mark a proper comparison image and region as part of agraphical marker. Thus, the learning engine 110 may be taught not justto interpret a single image or exam but also how to best selectcomparison images and what portion of those images are to be used forcomparison.

For example, the learning engine 110 may be configured to perform themachine learning to develop the model using the training information byevaluating a first portion designated by a first graphical marker withina first image included in the training information on a second imageincluded in the training information, wherein the second image wasacquired using a different imaging condition than the first image. Forexample, the second image may be a different image within the same examas the first image, a different exam than the exam including the firstimage, an image acquired at a different time after injection of an agentor isotope than the first image, an image acquired using a differentMill acquisition parameter than the first image, an image acquired usinga different radiation parameter than the first image, or a combinationthereof.

As noted above, machine learning generally refers to the ability of acomputer program to learn without being explicitly programmed and may beperformed using various types of methods and mechanisms. For example,machine learning may be performed using decision tree learning,association rule learning, artificial neural networks, inductive logicprogramming, support vector machines, clustering, Bayesian networks,reinforcement learning, representation learning, similarity and metriclearning, sparse dictionary learning, and genetic algorithms.

In some embodiments, the learning engine 110 assigns a weight to thetraining information or a portion thereof. For example, the learningengine 110 may be configured to obtain clinical results obtained aftergeneration of an image included in the training information and performa comparison of the clinical results and to the diagnostic informationassociated with the image (i.e., also included in the traininginformation). The learning engine 110 may then assign a weight to thediagnostic information based on the comparison, wherein the learningengine 110 develops the model based on the weight. In some embodiments,the clinical results include at least one image, at least one laboratoryresult, or a combination thereof.

Similarly, in some embodiments, after developing a model using thetraining information, the learning engine 110 may update the model basedon feedback designating a correctness of the training information or aportion thereof (e.g., diagnostic information included in the traininginformation). For example, in some embodiments, the learning engine 110updates a model based on clinical results (e.g., an image, a laboratoryresult, or the like) associated with one or more images included in thetraining information. In other embodiments, a user may manually indicatewhether diagnostic information included in the training information wascorrect as compared to an additional (e.g., a later-established)diagnosis. Further details are provided below regarding uses offeedback.

When the learning engine 110 develops one or more models (i.e., when thelearning engine 110 is considered trained), the models may be used toautomatically analyze images. For example, FIG. 5 illustrates a method500 performed by the server 102 (i.e., the electronic processor 104executing instructions, such as the learning engine 110) forautomatically analyzing clinical images using image analytics developedusing graphical reporting associated with previously-analyzed clinicalimages according to some embodiments.

As illustrated in FIG. 5, the method 500 includes receiving, with thelearning engine 110, training information from at least one data source112 over an interface (e.g., the input/output interface 108) (at block502) and performing machine learning to develop a model using thetraining information (at block 504). As described above with respect toFIG. 2, the training information includes a plurality of images andgraphical reporting associated with each of the plurality of images. Asalso described above, graphical reporting includes a graphical markerdesignating a portion of an image and diagnostic information associatedwith the marked portion of the image.

After the model is developed, the method 500 includes receiving, withthe learning engine 110, an image for analysis (at block 506). Thelearning engine 110 may receive the image for analysis from one or moreof the data sources 112. In other embodiments, the learning engine 110may receive the image for analysis through a browser application (e.g.,uploaded to a web page hosted by the server 102 or another device). Insome embodiments, the learning engine 110 also receives supplementaldata associated with the image for analysis, such as metadata associatedwith the image. The metadata may include order information, patientinformation (e.g., demographic information, historical information,family history information, and the like), modality information,procedure information, EMR information, laboratory information, publicinformation (e.g., data from the Center of Disease Control or otherpublic health organizations), and the like. The learning engine 110 mayrequest this supplemental data from the data sources 112 or otherdevices, such as through the communication network 111. For example, insome embodiments, the learning engine 110 may request order informationfor the image, which the learning engine 110 uses to identity additionalsupplemental data that may be useful for analyzing the image (e.g.,medical history information, patent demographic information, and thelike). The learning engine 110 may use the supplemental data todetermine what models should be used to process the image. For example,the learning engine 110 may process images associated with detectingbreast cancer using different models than images associated withdetecting fractures.

As illustrated in FIG. 5, the method 500 also includes automaticallyprocessing, with the learning engine 110, the received image using themodel to generate a diagnosis (i.e., a result) for the image (at block508). The learning engine 110 may output or provide a diagnosis invarious forms. For example, in some embodiments, the learning engine 110outputs a signal to a peripheral device, such as a display device (e.g.,monitor, touchscreen, and the like), a printer, and the like, thatincludes the diagnosis. For example, the learning engine 110 maygenerate a graphical user interface that includes the diagnosis, whereinthe graphical user interface is output to and displayed on a monitor,touchscreen, or other display device. The monitor may be included inperipheral device communicating with the server 102 or a computingdevice communicating with the server 102, such as a laptop computer,desktop computer, tablet computer, smart phone, smart watch or otherwearable, smart television, and the like. Also, in some embodiments, thelearning engine 110 may generate an electronic notification, such as ane-mail message, a Direct protocol message, or the like, that includesthe diagnosis. The learning engine 110 may transmit the electronicnotification to an email server, a HIS, a EMR, or another system. Also,in some embodiments, the learning engine 110 may store the diagnosis tomemory (e.g., local to the device performing the image analysis or aseparate device) for later retrieval.

In some embodiments, the learning engine 110 may incorporate thediagnosis into radiologist workflow, such that the diagnosis ispresented on a radiologist's workstation. For example, the learningengine 110 may be configured as a vendor neutral system (i.e., theserver 102 may interact with systems, such as RIS, HIS, PACS, and thelike, provided by numerous different providers) that communicates with areading workstation optimized to accept and use information from theserver 102. For example, the workstation may be configured to presentradiologists with annotated images generated by the learning engine 110in a format that allows the radiologist to edit the images, includingthe annotations, within a viewer (e.g., through a medical imageannotation tool). In some embodiments, edits made by the radiologist tothe annotated images are fed back to the learning engine 110, which usesthe edits to improve the developed models. Similarly, in someembodiments, the learning engine 110 may also be configured to generatea diagnosis that includes a measurement associated with the image andthe measurements may be displayed when the images are displayed (e.g.,as an overlay). For example, the learning engine 110 may be configuredto determine the volume of a lesion, which may be displayed with animage. Also, in some embodiments, the learning engine 110 may beconfigured to automatically determine and display a reconstructionimaging plane that best shows a particular measurement, such as themaximum diameter of a volumetric lesion.

Similarly, the learning engine 110 may be configured to output thediagnosis in an interoperable manner (e.g., to a PACS or a reportingsystem) such that reports (e.g., structured DICOM reports) arepre-populated with the diagnosis. In some embodiments, a user mayconfigure rules to designate what diagnoses generated by the learningengine 110 are mapped to particular data fields of one or more reports.This reporting process may leverage standards, such as the Annotationand Image Markup Standard, American College of Radiology (“ACR”) Assist,or the Integrating the Healthcare Enterprise's (“THE's”) Management ofRadiology Report Templates (“MRRT”) Profile. In addition, the reportingprocess may be interoperable with existing or emerging industrystandards for marking images, such as the Annotation and Image Markupstandard. Interoperability may enable a wide variety of PACS users toprovide close-loop feedback as described above, which expands thepopulation feeding training information to the learning engine 110.

Diagnoses generated by the learning engine 110 and included in reportsmay be approved by a diagnosing physician and user-specific,role-specific, organization-specific, facility-specific,region-specific, modality-specific, or exam-specific rules may be usedto control the approval process. For example, a PACS or similarreporting system may present a pre-processed report to a diagnosingphysician and allow the diagnosing physician to edit or approve specificsections of the report or the report in its entirety (e.g., via an inputaction, such as selection of a button, checkbox, radio button, and thelike and/or an electronic signature). As noted above, modifications madeto the report may be returned to the learning engine 110 eitherautomatically (e.g., by a rule) or manually on an ad hoc basis forcontinued refinement of the models.

The diagnosis may take many different forms, including for example, anindication of particular abnormality or anomaly (or the lack thereof)(e.g., cancer, fracture, blockage, bleeding, and the like), anindication of a disease, condition, or syndrome, a measurement ormorphological characteristic, an indication of an anatomical structure,or a combination thereof. For example, in some embodiments, thediagnosis generated by the learning engine 110 may include aclassification of an image into one of a plurality of categories, suchas: (1) normal (or normal with regard to a particular characteristic,such as a fracture not being present), (2) abnormal (or abnormal withregard to a particular characteristic, such as a fracture beingpresent), and (3) indeterminate. These classifications may be used invarious medical situations. For example, a categorization generated bythe learning engine 110 may be used to provide immediate diagnoses. Inparticular, a patient may take a picture of a skin lesion (e.g., using asoftware application on a mobile device, such as a smartphone or smartwearable), submit the image to the learning engine 110 for analysis(e.g., over the Internet), and receive an immediate diagnosis regardingwhether the lesion is malignant melanoma (or a recommendation that thepatient see a dermatologist or other specialist). Similarly, thelearning engine 110 may automatically analyze emergency room radiographsand automatically identify images showing a specific abnormality (e.g.,a fracture). The learning engine 110 may flag the identified images toensure that a patient is not inappropriately sent home without propertreatment (e.g., sent home with an untreated fracture or untreatedinternal bleeding). Also, in some embodiments, the learning engine 110may triage patients based on the analyzed images.

The learning engine 110 may also categorize images to flag imagesneeding manual review. For example, the learning engine 110 may labelimages stored in a PACS system as “normal,” “abnormal,” or“indeterminate,” and a diagnosing physician or other healthcareprofessional may use this information to triage images (e.g., prioritizeimages categorized as “abnormal” or “indeterminate”). For example, whenan image study including a plurality of images is submitted to thelearning engine 110 for analysis, the learning engine 101 may flag atleast one of the plurality of images included in the image study formanual review based on the categorization of each image of the pluralityof images. Similarly, the learning engine 110 may automatically identifyimages or exams that need to be sent to an external reading service. Inaddition, in some embodiments, the learning engine 110 automaticallyidentifies images as reference images that may be stored in variousrepositories and used for research, teaching, marketing, patienteducation, public health, or other purposes.

In some embodiments, the learning engine 110 may also determine aprobability for a diagnosis (e.g., a probability that the associateddiagnosis is correct). The probabilities may be used similar to thecategories described above (e.g., to triage images for review or toprovide immediate diagnoses). In some embodiments, the learning engine110 may also generate a basis for a probability (e.g., “The lesion has asmooth border; therefore, there is a low probability that the lesion ismalignant.”). In some situations, the learning engine 110 may generatemultiple diagnoses and may associate a probability with each diagnosisthat may be displayed to a user (e.g., ranked) to provide the user withpossible diagnoses and their associated probabilities as a differentialdiagnosis.

In some embodiments, the learning engine 110 may use thresholds tocategorize images as described above. For example, the learning engine110 may identify a value for an image (e.g., a measurement, aprobability, or the like) and may compare the determined value to one ormore thresholds to identify what category the image belongs to. In someembodiments, users can configure (i.e., adjust) the thresholds. Forexample, a user may adjust a threshold to include more or fewer imageswithin a particular category. As described below, these types ofadjustments (e.g., preferences or rules) may be set at an individuallevel or a group level.

The learning engine 110 may also use the categories or associatedthresholds to take particular automatic actions. For example, thelearning engine 110 may determine a value for an image and compare thevalue to one or more thresholds. When the value satisfies a particularthreshold (e.g., an accuracy threshold or a physical measurement orcharacteristics threshold, such as the length of size of a fracture),the learning engine 110 takes one or more automatic actions, such asdischarging a patient, scheduling a procedure for the patient, placingan order for a procedure (e.g., an imaging procedure, a laboratoryprocedure, a treatment procedure, a surgical procedure, or a combinationthereof), and the like. Similarly, the learning engine 110 may beconfigured to automatically provide recommendations based on determinedcategories or associated thresholds. For example, the learning engine110 may generate a report that states when follow-ups are needed orrecommended and what type of follow-up should be performed for aparticular diagnosis (e.g., “Pre-test probability of a breast cancer inthis patient is 1%. Based on image and clinical analytics, the post-testprobability has increased to 1.8%. Consider 6 month follow-upmammogram.”). The follow-ups may include an imaging procedure, alaboratory procedure, a treatment procedure, a surgical procedure, anoffice visit, or a combination thereof.

The learning engine 110 may also incorporate a diagnosis into one ormore reports (e.g., structured reports, such as DICOM structuredreports). The reports may include text, annotated images, flow charts,graphs, image overlays, image presentation states, or other forms. Forexample, as described above, a diagnosis determined by the learningengine 110 may be mapped to a particular data field of a structuredreport that is used to automatically pre-populate at least a portion ofthe structured report. As also described above, a diagnosing physicianor other healthcare professional may review these resulting reports orportion thereof and make modifications as necessary. The learning engine110 may use the modifications as a feedback loop that accelerates andimproves machine learning. For example, the learning engine 110 maydetect an abnormality in an image that a diagnosing physician disagreeswith or vice versa, and the learning engine 110 may use this feedback tofurther refine the developed models.

Also, in some embodiments, the diagnosis includes an image annotation,such as a graphical marker as described above. For example, the learningengine 110 may automatically add one or more graphical markers to animage that specify relevant areas of an image for manual review. In someembodiments, these marked images may be included in or associated with acorresponding report, such as a DICOM structured report.

In some embodiments, the diagnosis generated by the learning engine 110includes both an annotated image and corresponding diagnosticinformation. For example, the learning engine 110 may use the models toautomatically select relevant exams or images for comparison, includingdetermining advantageous comparison time frames. The learning engine 110may select comparison images using a patient's medical historyinformation, such as the timing of prior surgical or medical therapy.For example, by simultaneously weighing many factors, the learningengine 110 may automatically mark a lesion on an image and generate acorresponding report or notification that indicates influences on themarked lesions, such as “Lesion has decreased in size since date X;likely reasons include surgical debulking on date Y, and chemotherapy ondate Z.” Accordingly, in some embodiments, the learning engine 110determines an amount of change between two or more images and notifiesusers of a prior procedure or treatment potentially impacting anatomyrepresented in an image. The notifications may include a graphicalindication or a textual indication. For example, as described above, thelearning engine 110 may generate a textual description of priorprocedures or treatment potentially impacting imaged anatomy.Alternatively or in addition, the learning engine 110 may add agraphical mark (e.g., an icon) to an image that represents a priorprocedure or treatment (e.g., generally or a specific type of priorprocedure or treatment). When a graphical indication is used, a user mayselect (e.g., click on) the graphical indication to view the relevantmedical history information. Similarly, in some embodiments, thelearning engine 110 may generate a data structure that tracks changes ina detected anomaly over a sequence of images. The learning engine 110may cross-reference changes stored in the data structure with at leastone medical procedure or treatment performed on a patient to allow auser to identify the results (or lack thereof) of the procedure ortreatment represented within images. For example, in some embodiments,the learning engine 110 may be configured to automatically register andcompare serial exams to make comparisons and report differences. Inparticular, the learning engine 110 may be configured to automaticallygraph or otherwise access changes in a lesion or lesions over a periodof time and issue a report using a standardized methodology, such as aRECIST 1.1 table.

Similarly, the learning engine 110 may use medical history informationof a patient, such as clinical notes to automatically determine aportion of an image of clinical concern. For example, when a patientarrives at an emergency room and indicates that he struck his thumb witha hammer, this information (i.e., a patient complaint, an anatomicalstructure associated with the complaint, and the like) may be stored asclinical notes (e.g., in a patient's EMR). The learning engine 110 mayuse these clinical notes when processing x-ray images of the patient'shand to determine one or more portions of the image that may be ofclinical concern. In particular, the learning engine 110 may determine aportion of the x-ray image region representing the patient's thumb, or,in particular, a tip of the patient's thumb. The learning engine 110 mayindicate the determined portion using a textual indication or agraphical indication that is displayed with the x-ray image or on thex-ray image. Accordingly, when a diagnosing physician reviews the x-rayimage, the indication directs the diagnosing physician's attention tothe relevant portion of the x-ray image (i.e., the thumb or the tip ofthe thumb). Thus, the diagnosing physician is better able to diagnosethe x-ray image. In some embodiments, the learning engine 110 mayautomatically learn these correlations based on training informationthat includes marked areas of images and corresponding clinical notes.Furthermore, as described above, in some embodiments, the learningengine 110 learns to identify anatomical structures within an imageusing the training information. Accordingly, the learning engine 110 mayidentify an anatomical structure based on the clinical notes associatedwith an image submitted for analysis and then automatically identify andmark the corresponding anatomical structure in the associated image.

In some applications, the models developed by the learning engine 110may be used when a diagnosing physician is not available, such as duringemergency, disaster, and military situations or in rural areas wherediagnosing physicians are not readily available. Similarly, the modelsmay be used to automatically identify what image studies should be sentto an expert reading service. Also, in some situations, the models maybe used in place of a diagnosing physician, such as in reading ECG data,where the models may provide a high level of accuracy akin to that of adiagnosing physician. Also, as described in more detail below, themodels may be “tuned” to a particular user (e.g., a diagnosing physicianor other healthcare provider), facility, organization, nation,geographic region, or season of the year. For example, the learningengine 110 may generate a chest radiograph report that includes astatement such as, “Pulmonary hazy opacity. Consider flu season andlocally reported SARS epidemic.”

Accordingly, the models developed using the training informationdescribed above may be used for many purposes, including, for exampleand without limitation, automatically identifying lesions as suspiciousfor melanoma based on skin images (e.g., photographs), automaticallyreading ECG data to a greater accuracy than a human diagnosing physicianor eliminating the need for human review of some ECG data, eliminatingthe need for physicians to view mammogram exams or images, drivingphysicians to more carefully inspect some images of a CT exampreferentially over others, identifying patients with intracranialhemorrhages on a CT scan performed in emergency room settings or othersettings, performing automatic review of images obtained via endoscopy,colonoscopy, or CT colonography, automatically detecting nodules on CTscans that are suspicious for lung cancer as part of a screeningprogram, automatically detecting compression fractures in lumbar imagingexams, and automatically detecting spinal stenosis or abnormal discs inlumbar imaging exams.

In some embodiments, the learning engine 110 may also be configured todetermine when additional information is needed or beneficial togenerate a diagnosis. For example, the learning engine 110 may havecertain diagnoses (e.g., differential diagnoses) that the learningengine 110 knows can be refined with certain additional information,such as an additional image, patient demographic information, medicalhistory information, a laboratory result, and the like. Accordingly,when the learning engine 110 receives an image for analysis, thelearning engine 110 may determine whether additional information isneeded (e.g., using the models). When the additional information isneeded, the learning engine 110 may ask a user whether he or she wantsto provide the additional information to refine the diagnosis (e.g., bydisplaying an icon that the user can select or providing anotherprompt). In some embodiments, the learning engine 110 may also informthe user what types of additional information (e.g., how many questionsmay be asked) may be requested and what impact the additionalinformation may have (e.g., eliminate one or ten potential diagnoses).Alternatively or in addition, a user may configure the learning engine110 to automatically request the additional information. When thelearning engine 110 receives the requested additional information, thelearning engine 110 generates a diagnosis as described above using themodels and the additional information. For example, when the learningengine 110 generates an initial diagnosis that indicates a rare diseasecommonly only reported in African countries, the learning engine 110 maybe configured to request additional information regarding whether thepatient recently traveled outside of the U.S. or interacted with otherswho received traveled outside of the U.S. to determine the probabilityof the initial diagnosis. Similarly, when the learning engine 110detects air within an abdomen represented in an image, the learningengine 110 may ask a user (e.g., a nurse) whether the patient associatedwith the image has undergone any surgical or other procedures in thepast (e.g., the past ten days) that would explain the detection of air.

In some embodiments, the learning engine 110 prompts one or more users(e.g., the patient, one or more users associated with the care of apatient, or a combination thereof) for the additional information,automatically accesses the additional information from at least oneelectronic data repository (e.g., a HIS, an EMR, or the like), orperforms a combination thereof. The learning engine 110 may prompt auser for the additional information within an image review application.Alternatively or in addition, the learning engine may prompt a user forthe additional information by transmitting an electronic message to theuser, such as a page, an e-mail message, a Direct protocol message, atext message, a voicemail message, or a combination thereof. In someembodiments, a user may define one or more preferences for theseprompts. The preferences may specify when a user is prompted foradditional information, how the user is prompted for the additionalinformation (e.g., page followed by email or only email once a day), ora combination thereof. For example, the preferences may include athreshold associated with an initial diagnosis, and the learning engine110 may prompt the user for additional information only when aprobability associated with an initial diagnosis (i.e., a diagnosisdetermined without the additional information) is less than thethreshold. As another example, the preferences may include a thresholdassociated with an updated or refined diagnosis. For example, thethreshold may specify an amount of change (i.e., in a probability) thatmust be associated with the additional information before the learningengine 110 prompts a user for additional information. As one example, auser may specify that he or she only wants to be prompted for additionalinformation when the additional information increases the probability ofthe initial diagnosis by at least 20%. Accordingly, when an initialdiagnosis has a probability of 40% and this probability would increaseto 60% if the additional information were available (e.g., a weight ofthe patient, a blood pressure of the patient, and the like), thelearning engine 110 prompts the user for the additional information.Conversely, when an initial diagnosis has a probability of 30% and thisprobability would increase to 40% if the additional information wereavailable, the learning engine 110 does not prompt the user.Furthermore, in some embodiments, the learning engine 110 may prompt auser regarding whether the additional information should be obtained.Similarly, in some embodiments, the learning engine 110 directly promptsa user for the additional information. In other embodiments, thelearning engine 110 prompts a user for a source of the additionalinformation (e.g., the name of another user or a system storing theadditional information).

For example, in some embodiments, the learning engine 110 may generate aset of initial diagnoses (e.g., possible diagnoses) for an image. Basedon the set of initial diagnoses, additional information may bedetermined that would potentially reduce the number of initial diagnoses(e.g., to provide a single or most likely diagnosis). For example, whenan initial diagnosis includes a possible syndrome, the learning engine110 may use additional information to confirm or eliminate the possiblesyndrome. In particular, the learning engine 110 may prompt a userregarding other detected abnormalities, such as “Are there renalabnormalities?” In some embodiments, the learning engine 110 may alsospecify a basis for the requested additional information, such as “Ifrenal abnormalities exist, consider Von Hippel Lindau syndrome.”Accordingly, a user may understand why the additional information isbeing requested.

After the additional information is received, the learning engine 110updates the set of initial diagnoses based on the additionalinformation, such as by reducing the number of diagnoses included in theset or updating a probability of one or more diagnoses included in theset. The set of initial diagnoses, the updated set of diagnoses, or bothmay be displayed to a user (e.g., with the associated probabilities andoptional bases) as a diagnosis. This process may also be repeated (e.g.,automatically or in response to a user request) to further refine adiagnosis.

Similarly, in some embodiments, the learning engine 110 may beconfigured to determine when additional imaging may aid a diagnosis. Forexample, FIG. 6 illustrates a method 600 performed by the server 102(i.e., the electronic processor 104 executing instructions, such as thelearning engine 110) for automatically analyzing clinical images anddetermining when additional imaging may aid a diagnosis according tosome embodiments.

As illustrated in FIG. 6, the method 600 includes receiving, with thelearning engine 110, an image of a patient from at least one data source112 over an interface (e.g., the input/output interface 108) (at block602). The method 600 also includes automatically processing the image,with the learning engine 110, to generate a diagnosis for the image (atblock 604). In some embodiments, the learning engine 110 generates thediagnosis as described above with respect to FIG. 5.

The learning engine 110 also automatically determines, based on thediagnosis, at least one additional image for supplementing the diagnosis(at block 606) and automatically locates the at least one additionalimage for the patient (at block 608). For example, when the initialdiagnosis is inconclusive findings, the learning engine 110 may applycriteria, such as Appropriate Use Criteria (“AUC”), to determine whatother type of imaging is recommended and then query one or more systemsstoring the determined type of imaging. In some embodiments, thelearning engine 110 accesses one or more PACSs, vendor neutral archives,RISs, EMRs, HISs, image study ordering systems, computer assisteddetection (CAD) systems, or a combination thereof to locate theadditional image. Alternatively or in addition, the learning engine 110may prompt a user for the additional image (e.g., a storage location orsource of the additional image or an indication of whether theadditional image has been or will be performed or ordered).

The additional image may be generated before or after the image. Whenthe additional image has not already been performed for a patient, thelearning engine 110 may put the diagnosis on hold until the additionalimage is available. For example, the learning engine 110 may monitorsources of the additional image to detect when the additional image isavailable. The learning engine 110 may also generate notifications whenthe additional image remains unavailable after a configurable amount oftime. The notifications may be transmitted to one or more users, such asthe patient's physician who ordered the initial image. Also, in someembodiments, the learning engine 110 may take one or more automaticactions to obtain the additional image, such as automatically placing anorder for the additional image. In some embodiments, the learning engine110 waits a configurable amount of time before taking any automaticactions.

The at least one additional image may be generated by an imagingmodality different than the original image (e.g., original x-ray imagefollowed by a CT scan or MRI). Alternatively or in addition, the atleast one additional image may be generated using an imaging parameterdifferent than the original image. As example only, the differentimaging parameter may be whether a contrast agent was used, whether aradioactive isotope was used, a time after an agent or isotope wasintroduced, a MRI pulse sequence, a CT radiation level, or a combinationthereof. Also, in some embodiments, the additional image may beperformed for a portion of a patient that is the same or different thanthe portion of the patient represented in the initial image. Forexample, when an initial diagnosis associated with the initial imageeffects multiple anatomical structures, the learning engine 110 may usethe additional image to determine whether other anatomical structures ofthe patient (i.e., different from the anatomical structure representedin the initial image) supports or contrasts the potential diagnosis.

As illustrated in FIG. 6, when the additional image is available, thelearning engine 110 automatically updates the initial diagnosis based onthe additional image (at block 610). In some embodiments, the updateddiagnosis includes a differential diagnosis (e.g., a diagnosisdistinguishing a particular disease or condition from others thatpresent similar symptoms). In this respect, the learning engine 110 usesthe additional image to eliminate some potential diagnoses.

As noted above, in some embodiments, the learning engine 110 isconfigured to determine relevant images within an image study, relevantportions of an image, or a combination thereof. For example, thelearning engine 110 may select a subset of images included in an imagestudy for display to a physician that provide the best view of aparticular portion of anatomy or a particular abnormality. For example,the learning engine 110 may identify a multi-planar or a 3-Dpresentation of a data set (e.g., a volume CT, MRI, PET, or ultrasounddata set) that shows an aspect of the anatomy or pathology at apreferred or optimal vantage for a diagnosing physician (e.g., based ona suspected or prior diagnosis). Similarly, within a CT examinationincluding 1000 images, the learning engine 110 may identify 20 imagesthat warrant manual review and may flag these images accordingly. Insome embodiments, the learning engine 110 identifies these optimalimages or views based on received training information as describedabove that includes graphical reporting for particular images or views.

Similarly, the learning engine 110 may annotate images to highlightportions of images that the learning engine 110 identifies as being most“important” for manual review. The learning engine 110 may selectrelevant images or portions of images based on exam indications orprobabilities of abnormal findings and common locations for suchabnormal findings. Similarly, the learning engine 110 may mark the rightparatracheal region on a chest radiograph when abnormalities in thisregion are frequently missed or when abnormalities likely occur in thisregion.

For example, FIG. 7 illustrates a method 700 performed by the server 102(i.e., the electronic processor 104 executing instructions, such as thelearning engine 110) for automatically determining clinical imageswithin an image study for display to a diagnosing physician according tosome embodiments. As illustrated in FIG. 7, the method 700 includesreceiving, with the learning engine 110, training information from atleast one data source 112 over an interface (e.g., the input/outputinterface 108), wherein the training information includes a plurality ofimage studies (at block 702). The method 700 also includes determining,with the learning engine 110, a subset of images included in each of theplurality of image studies displayed to one or more diagnosingphysicians (at block 704) and performing, with the learning engine 110,machine learning to develop a model based on the training informationand the subset of images included in each of the plurality of imagestudies (at block 706). In some embodiments, the learning engine 110develops the models as described above with respect to FIGS. 2 and 5.

In some embodiments, the learning engine 110 determines subset of imagesincluded in the training information that were displayed to a diagnosingphysician based on what images included in an image study wereassociated with an annotation. The annotation may include a graphicalmarker designating a portion of an image as described above. In otherembodiments, the annotation may include other forms of diagnosticinformation associated with an image. Accordingly, in some embodiments,the determined subset of images included in an image study includes atleast two images of the same anatomical structure of a patient obtainedat different times during an imaging procedure. The learning engine 110may store image characteristics for the determined subset of images in adata structure and may use the data structure (e.g., as part ofdeveloping the model) to identify image characteristics associated withimages commonly reviewed by a diagnosing physician (i.e.,clinically-relevant images). Accordingly, the learning engine 110 mayidentify a subset of images within an image study having imagecharacteristics that help a diagnosing physician review the image studyefficiently and effectively.

In particular, as illustrated in FIG. 7, after developing the model (atblock 706), the method 700 includes receiving, with the learning engine110, an image study including a plurality of images (at block 708) andprocessing the image study, with the learning engine 110, using themodel to determine a subset of the plurality of images included in theimage study (at block 710). The learning engine 110 flags the determinedsubset of the plurality of images included in the image study for manualreview by a diagnosing physician (at block 712).

In some embodiments, the flagged subset is used when the image study isopened or displayed within an image review application (e.g., by adiagnosing physician). For example, when the image study is displayedwithin the image review application, the flagged subset may begraphically marked (e.g., using an icon or other distinguishingfeature). Also, in some embodiments, the flagged subset of the pluralityof images may be displayed within the image review application beforedisplaying a remainder of the plurality of images included in the imagestudy. Alternatively, only the flagged subset of the plurality of imagesmay be displayed within the image review application.

In addition, in some embodiments, the learning engine 110 assigns apriority to images included in a flagged subset. The priority may bebased on the frequency at which images with particular imagecharacteristics are displayed to a diagnosing physician. For example,the learning engine 110 may assign an image having image characteristicsthat are displayed 90% of the time (as learned from the traininginformation) a higher priority than an image having imagecharacteristics that are displayed 70% of the time. When the image studyis displayed within an image review application, the images included inthe flagged subset may be displayed in an order based on their assignedpriorities (e.g., higher priority images displayed before lower priorityimages). Similarly, in some embodiments, only images with a priorityabove a configurable threshold may be initially displayed or displayedat all. In some embodiments, the learning engine 110 may similarlyassign a priority to an overall image study based on the flagged subsetof images (e.g., as a composite priority based on the priorities forindividual images, as a priority based on a number of images included inthe flagged subset, and the like). The priority assigned to the imagestudy may be used to schedule an image study for manual review. Forexample, an image study with fewer flagged images may be easier orharder to complete than an image study with more flagged images. Also,in some embodiments, the priorities assigned to image studies may beused to schedule the studies based on the work schedule of a diagnosingphysician. For example, diagnosing physicians may be more focused atcertain times of the day and, hence, image studies with prioritiessatisfying a particular threshold (e.g., greater than or less than athreshold) may be schedule during this time if they likely require moremanual review than other image studies.

The learning engine 110 may also be configured to perform a similarmethod to determine portions of a single image to display to adiagnosing physician. For example, FIG. 8 illustrates a method 800performed by the server 102 (i.e., the electronic processor 104executing instructions, such as the learning engine 110) forautomatically determining portions of clinical images for display to adiagnosing physician according to some embodiments.

As illustrated in FIG. 8, the method 800 includes receiving, with thelearning engine 110, training information from at least one data source112 over an interface (e.g., the input/output interface 108) (at block802). The training information includes a plurality of images (at block802). The method 800 also includes determining, with the learning engine110, a portion of each of the plurality of images including diagnosticinformation (at block 804) and performing, with the learning engine 110,machine learning to develop a model using the training information andthe portion determined for each of the plurality of images (at block806). As described above with respect to the method 700, the learningengine 110 may determine the portion of each of the plurality of imagesbased on an annotation (e.g., a graphical marker) included in an image.

The method 800 also includes receiving, with the learning engine 110, animage for analysis (at block 808), processing, with the learning engine110, the image using the model to automatically determine a portion ofthe image for manual review (at block 810), and flagging, with thelearning engine 110, the portion of the image (at block 812). In someembodiments, multiple portions of an image may be flagged or no portionsof an image may be flagged (e.g., when an image does not include arelevant portion).

As described above with respect to the method 700, the flagged portionof the image may be used when the image is displayed within an imagereview application. For example, an image review application maygraphically mark the flagged portion of the image when the image isdisplayed (e.g., using a graphical marker as described above), maypreferentially display the flagged portion of the image when the image(e.g., centered within a window displaying the image), or may displaythe image with the flagged portion of the image in a preferential formatas compared to an image without a flagged portion. Also, in someembodiments, the learning engine 110 assigns a priority to an imagebased on the flagged portion of the image. The priority may designate anorder of display of the images within an image review application.Similarly, the priority assigned to an image may be used to schedule theimage for manual review (or schedule the image study including theimage). Also, when the image is included in an image study, the imagewith the flagged portion may be prioritized (e.g., displayed before)other images included in the image study (e.g., images not including aflagged portion) when the image study is displayed for review.

In some embodiments, the learning engine 110 may apply rules whenprocessing images. For example, as noted above, the learning engine 110may develop different models for different types of anatomy. Thelearning engine 110 may similarly develop different models for differenttypes of images (e.g., ECG data, ultrasounds, CT scans, sonograms, andthe like). Accordingly, the learning engine 110 may apply rules topre-process a received image to identify what models should be appliedto the image. For example, in some embodiments, the learning engine 110may pre-process a received image using associated clinical andhistorical information to identify what abnormalities may be found inthe image.

In some embodiments, a user may set or modify the rules applied by thelearning engine 110. For example, a physician may be able to specifywhat particular models should be applied to a particular type of image.For example, when a CT scan is performed, the learning engine 110 may beconfigured (e.g., through a rule) to check for internal bleeding andprovide a “yes” or “no” diagnose (e.g., in addition to performing othertypes of diagnoses relevant for a CT scan and the particular type ofanatomy captured in the image). Similarly, the learning engine 110 maybe configured (e.g., through a rule) to focus on a particular portion ofan image or a particular type of diagnosis that is commonly missedduring manual review. These rules may be set at a particular user level(e.g., the physician that submitted the image for automatic processing,the diagnosing physician, the patient, and the like), a clinic level(e.g., all images submitted from a particular hospital or imagingcenter), a network level (e.g., all images submitted from hospitals andimaging centers associated with a particular health care network orservice provider, such as Anthem or Athena), or a system-wide level(e.g., all images submitted for automatic analysis).

For example, FIG. 9 illustrates a method 900 performed by the server 102(i.e., the electronic processor 104 executing instructions, such as thelearning engine 11) for automatically analyzing clinical images usingrules and image analytics developed using graphical reporting associatedwith previously-analyzed clinical images according to some embodiments.As illustrated in FIG. 9, the method 900 includes receiving, with thelearning engine 110, training information from at least one data source112 over an interface (e.g., the input/output interface 108) (at block902). As described above with respect to methods 200 and 500, thetraining information includes a plurality of images and graphicalreporting associated with each of the plurality of images. The graphicalreporting includes a graphical marker designating a portion of an imageand diagnostic information associated with the marked portion of theimage. As illustrated in FIG. 9, the method 900 also includesperforming, with the learning engine 110, machine learning to develop amodel using the training information (at block 904).

The method 900 also includes receiving, with the learning engine 110, animage for analysis (at block 906), determining, with the learning engine110, a set of rules for the image (at block 908), and automaticallyprocessing, with the learning engine 110, the image using the model andthe set of rules to generate a diagnosis for the image (at block 910).As described above, the learning engine 110 may determine the set ofrules based, for example, on a user viewing the image within an imagereview application, a diagnosing physician performing a diagnosis of theimage, a physician ordering an image procedure during which the imagewas generated, an organization of diagnosing physicians associated withthe image, an organization of healthcare facilities associated with theimage, an organization associated with a workstation displaying theimage, an imaging modality generating the image, or an image acquisitionsite associated with the image. Alternatively or in addition, thelearning engine 110 may determine the set of rules based on patientdemographic information associated with the image, a type of the imageor a type of an imaging exam associated with the image, a geographiclocation associated with the image (e.g., a location where the image wastaken or a location of the patient), or a geographical location of adiagnosing physician associated with the image. Also, in someembodiments, the learning engine 110 may determine the set of rulesbased on applicable epidemic information. Furthermore, in someembodiments, the learning engine 110 determines an anatomical structurerepresented in the image and determines the set of rules based on theanatomical structure.

In some embodiments, the learning engine 110 automatically generates therules. For example, the learning engine 110 may use training informationassociated with a particular user, facility, organization, and the liketo automatically learn applicable rules. Alternatively or in addition,the learning engine 110 may receive manually-defined rules from a user.For example, in some embodiments, the learning engine 110 (or a separatedevice or piece of software) may be configured to generate a graphicaluser interface that displays a set of rules (whether automatically ormanually defined) and allows a user to modify the set of rules.

As described above, the learning engine 110 may update the developedmodels based on new training information, feedback regarding theperformance of the models, or a combination thereof. For example, aftergenerating a diagnosis for an image, the learning engine 110 may collectinformation from one or more sources (e.g., pathology reports or otherlaboratory reports) to identify whether the generated diagnosis wascorrect. When the diagnosis was not correct, the learning engine 110 mayupdate the one or more models used to generate the diagnosis (e.g., bymaking an adjustment to the model, by re-developing the model using theupdated training information, or by another technique for updatingmodels).

The learning engine 110 may similarly use this feedback to performoutcome analytics. For example, when pathology results are available forparticular images, the learning engine 110 may track how well the imageswere analyzed (e.g., automatically or manually by a diagnosingphysician). The learning engine 110 may use this information todetermine how well the models or a physician or a group of physiciansare performing particular diagnoses (e.g., detecting cancer). Forexample, the learning engine 110 may be configured to determine how wella particular physician or group of physicians is performing as comparedto other physicians, groups of physicians, or national or worldwideaverages.

The learning engine 110 may also be configured to use this informationto perform analytics to determine parameters influencing a correctdiagnosis or an incorrect diagnosis, such as a number and the types ofimages taken, the imaging equipment, and the like. For example, thelearning engine 110 may capture information from DICOM header files anddetect correlations between particular imaging parameters and correct orincorrect diagnoses (e.g., using a multivariate data analysis).

Similarly, the learning engine 110 may be configured to monitor andreport diagnostic patterns and trends. For example, one diagnosingphysician reading chest x-rays in an institution may diagnose congestiveheart failure at a greater incidence than another diagnosing physician.Accordingly, the learning engine 110 may use this information toidentify which diagnosing physician is correct, identify diagnosingphysicians needing further training or supervision, identify diagnosingphysicians to act as an “expert” for reviewing particular images (e.g.,images identified as being “indeterminate” by the learning engine 110),or identify training information that should be assigned a greaterweight by the learning engine 110. For example, in some embodiments, thelearning engine 110 may (e.g., automatically) schedule training for adiagnosing physician or a group of diagnosing physicians when anassociated score falls below a configurable threshold. The learningengine 110 may also use this information to provide closed feedbackloops to particular physicians or facilities. For example, the learningengine 110 may provide statistics such as “94% of chest x-rays orderedfrom this clinic this month were normal,” or “99% of your ordered anklex-rays showed no fracture, the average for this clinic is 85%,” whichphysicians and facilities may use to identify areas for improvement.Similarly, the learning engine 110 may be configured to detect a trend,such an increase in particular diagnoses, an identification of apopulation of patients associated with a particular diagnosis, anepidemic, or an underlying cause for a particular diagnosis (e.g.,Legionnaire's disease). Thus, the learning engine 110 may use feedbackto perform audits, determine quality of care, determine payment andreimbursement rates, allocate human resources, or a combination thereof.

For example, FIG. 10 illustrates a method 1000 performed by the server102 (i.e., the electronic processor 104 executing instructions, such asthe learning engine 110) for automatically scoring diagnoses associatedwith clinical images according to some embodiments. As illustrated inFIG. 10, the method 1000 includes receiving, with the learning engine110, a diagnosis associated with an image of a patient from at least onedata source over an interface (e.g., the input/output interface 108) (atblock 1002). The diagnosis relates to an anatomical structurerepresented in the image. The diagnosis may be a manual diagnosisgenerated by a diagnosing physician, an automatic diagnosis generated bya computer system (e.g., the learning engine 110), or a combinationthereof.

The method 1000 also includes receiving, with the learning engine 110, apathology result for the patient (e.g., from at least one pathologyresult source) for the anatomical structure over an interface (e.g., theinput/output interface 108) (at block 1004). In some embodiments, thepathology result was generated after the received diagnosis. The method1000 also includes automatically generating, with the learning engine110, a score based on a comparison of the diagnosis and the pathologyresult (at block 1006). The method 1000 may also include displaying,with the learning engine 110, the score within a graphical userinterface (at block 1008).

The learning engine 110 may generate a score for an individualdiagnosing physician, a plurality of diagnosing physicians, anindividual facility, a network of a plurality of facilities, or anotherconfigurable population. For example, in some embodiments, the learningengine 110 generates a score for itself or another computer systemconfigured to automatically generate diagnoses. In some embodiments, thelearning engine 110 displays a score within a graphical interface thatincludes a plurality of scores, such as a score for each of a pluralityof individual diagnosing physicians. In some embodiments, the learningengine 110 ranks the displayed scores (e.g., highest score to lowestscore).

The learning engine 110 may generate a score that includes a percentageof diagnoses not matching the corresponding pathology result, apercentage of diagnoses matching the corresponding pathology result, orboth. Also, in some embodiments, the learning engine 110 generates ascore that compares one diagnosing physician to another diagnosingphysician or to a base score (e.g., representing a normal or acceptableamount of error). For example, in some embodiments, the learning engine110 generates a score that compares the performance of a diagnosingphysician to performance of a computer system configured toautomatically generate diagnoses. The learning engine 110 may also beconfigured to generate a score for a particular type of imagingmodality, a particular type of the image or a particular type of imagingexam associated with the image, a particular geographic location, aparticular time of day, a particular type or population of patients, ora combination thereof. Also, in some embodiments, the learning engine110 may generate a score based on one or more configurable preferences(e.g., associated with a user, a group of users, or the like).

As noted above, when the original diagnosis was automatically generatedby a computer system, such as the learning engine 110, the computersystem may be updated based on the score. For example, the learningengine 110 may use a score to determine when to perform an updates.Similarly, the learning engine 110 may use a score to assign a weight toparticular training information as described above.

In some embodiments, the learning engine 110 may also be configured toaid a diagnosing physician in difficult activities, such as identifyingparticular abnormalities that are commonly missed by diagnosingphysicians. For example, when analyzing a submitted image, the learningengine 110 may be configured to screen chest x-rays and chest CT scansfor pulmonary nodules measuring less than a configurable size that arecommonly missed by diagnosing physician. Similarly, the learning engine110 may be configured to screen thyroid glands, skin, and subcutaneoustissues for abnormalities on a chest CT scan when diagnosing physicianscommonly miss lesions in these locations. As another example, thelearning engine 110 may be configured to focus on particular regions ofan image or particular abnormalities that a particular diagnosingphysician commonly misses (e.g., a known blind spots or bias of aparticular physician). In some embodiments, a diagnosing physician maymanually specify (e.g., through one or more rules) aids for particularactivities. However, in other embodiments, the learning engine 110 maybe configured to automatically identify common errors for individualphysicians or populations of physicians.

For example, FIG. 11 illustrates a method 1100 performed by the server102 (i.e., the electronic processor 104 executing instructions, such asthe learning engine 110) for automatically determining diagnosisdiscrepancies for clinical images according to some embodiments. Asillustrated in FIG. 11, the method 1100 includes receiving, with thelearning engine 110, a first diagnosis from at least one data source 112over an interface (e.g., the input/output interface 108) (at block1102). The first diagnosis relates to an anatomical structurerepresented in an image. In some embodiments, the first diagnosis is adiagnosis manually specified by a diagnosing physician. The method 1100also includes determining, with the learning engine, a second diagnosisfor the anatomical structure generated after the first diagnosis (atblock 1104). In some embodiments, the second diagnosis includes apathology report for the anatomical structure, a diagnosis associatedwith additional imaging of the anatomical structure (i.e., differentfrom the image associated with the first diagnosis), acomputer-generated diagnosis associated with an image (e.g., the sameimage associated with the first diagnosis or a different image), alaboratory finding, an electronic medical record (EMR), or a combinationthereof.

As illustrated in FIG. 11, the learning engine 110 stores the firstdiagnosis and the second diagnosis within a data structure (e.g., atable) (at block 1106) and uses the data structure to automaticallydetermine a discrepancy between the first diagnosis and the seconddiagnosis (at block 1108).

In some embodiments, the learning engine 110 may generate a report basedon the data structure. The report may be for the diagnosing physiciansor a plurality of diagnosing physician including the diagnosingphysician. The report may include the discrepancy, which may berepresented as an occurrence rate associated with the discrepancy (e.g.,how frequently a diagnosing physician provides a diagnosis that does notmatch with a later diagnosis). In some embodiments, the report isparsable (e.g., sortable, filterable, and the like), such as by animaging modality, a diagnosing physician, an exam type, an image type,an anatomical structure, patient demographic information, or a date.

In addition to or as an alternative to generating a report, the learningengine 110 may generate one or more electronic notifications based onthe data structure or a particular discrepancy identified based on thedata structure. The learning engine 110 may send the notification to oneor more users or systems, such as the diagnosing physician thatgenerated a first diagnosis. In some embodiments, the electronicnotification includes an e-mail message, a page, an instant message, orthe like. Also, in some embodiments, the learning engine 110 generatesand sends electronic notifications based on one or more configurablepreferences (e.g., of the diagnosing physician associated with thediscrepancy).

In addition to or as an alternative to generating a report or generatingelectronic notifications, the learning engine 110 may graphical mark aportion of the image associated with a first diagnosis based on thediscrepancy. For example, when a diagnosing physician misses an anomalywithin the image as part of the first diagnosis that is detected as partof the second diagnosis, the learning engine 110 may be configured toautomatically mark the missed anomaly within the image. By marking theanomaly, the diagnosing physician that missed the anomaly can review theimage to learn from his or her mistakes.

Similarly, in some embodiments, the learning engine 110 applies the datastructure to new images being reviewed by a diagnosing physician. Forexample, when a new image is received for review by a particulardiagnosing physician, the learning engine 110 may automaticallygraphically mark one or more portions of the new image based on the datastructure. The learning engine 110 may graphically mark the new image bymarking a detected anomaly within the new image, one or more portions ofthe new image that may potentially include the anomaly, or a combinationthereof. For example, when a particular diagnosing physician (or apopulation of diagnosing physicians) commonly misses anomalies in aparticular portion of a lung, the learning engine 110 may automaticallymark this portion on the image to draw the diagnosing physician'sattention to the portion. Accordingly, when a diagnosing physician isreviewing the new image to provide a diagnosis, the diagnosing physicianis alerted to previous discrepancies to mitigate future discrepancies.

In addition to or as an alternative to providing the graphical markings,the learning engine 110 may automatically display a warning with the newimage based on the data structure to alert the diagnosing physician ofprevious discrepancies to mitigate future discrepancies. The graphicalmarking, the warnings, or both may be configurable based on one or morepreferences of a user. For example, a user may be able to enable ordisable the graphical markings, the warnings, or both.

Similarly, as described above, a particular diagnosing physician may(subconsciously) develop a bias. For example, after diagnosing fivecases of breast cancer, a diagnosing physician may be reluctant to makea further diagnosis of breast cancer. Similarly, a particular diagnosingphysician may be less likely to correctly diagnosis a particularabnormality for a particular population of patients or under certainreading conditions.

Accordingly, FIG. 12 illustrates a method 1200 performed by the sever102 (i.e., the electronic processor 104 executing instructions, such asthe learning engine 110) for automatically determining a potential biasof a diagnosing physician according to some embodiments. As illustratedin FIG. 12, the method 1200 may include receiving, with the learningengine 110, a first diagnosis specified by the diagnosing physician fromat least one data source 112 over an interface (e.g., the input/outputinterface 108) (at block 1202). The first diagnosis may be associatedwith an anatomical structure represented in an image. The method 1200also includes determining, with the learning engine 110, a seconddiagnosis for the anatomical structure (at block 1204) and storing, withthe learning engine 110, the first diagnosis and the second diagnosis ina data structure (at block 1206). The learning engine 110 alsodetermines, based on the data structure, a discrepancy between the firstdiagnosis and the second diagnosis (at block 1208) and automaticallydetermines a correlation between the discrepancy and additionalinformation (at block 1201). As described in more detail below, thecorrelation may identify a potential cause of the discrepancy.

The additional information may include a time of day of the firstdiagnosis or a day of the week of the first diagnosis. Accordingly, thelearning engine 110 may determine a correlation between a bias of thediagnosing physician and a particular time of the day or day of theweek. Alternatively or in addition, the additional information mayinclude patient demographic information associated with the firstdiagnosis, such as, for example, a patient's age, race, or gender. Thus,the learning engine 110 may determine a correlation between a bias ofthe diagnosing physician and a particular diagnosis for a particulartype of patient. Alternatively or in addition, the additionalinformation may include at least one diagnosis specified by thediagnosing physician prior to the first diagnosis (e.g., during areading session including the first diagnosis or another configurableperiod of time prior to the first diagnosis). Accordingly, the learningengine 110 may determine a correlation between a bias of the diagnosingphysician and a particular reading schedule or sequence of priordiagnoses. For example, the additional information may include an amountof time between the first diagnosis and a third diagnosis specified bythe diagnosing physician prior to the first diagnosis matching the firstdiagnosis. Thus, the learning engine 110 may determine a correlationbetween a bias of the diagnosing physician and a prior frequency orsequence of similar diagnoses (e.g., less inclined to render an abnormaldiagnosis after five or more abnormal diagnoses were previouslyrendered).

The additional information may also include information associated withthe image used to generate the first diagnosis, such as an imagingmodality associated with the first diagnosis, or an imaging acquisitionsite associated with the first diagnosis. Accordingly, the learningengine 110 may determine a correlation between particular diagnosis andparticular imaging information. In other embodiments, the additionalinformation may include an anomaly or anatomical structure identified aspart of the first diagnosis or the second diagnosis that may represent abias of a diagnosing physician toward particular anomalies or particularanatomical structures. Similarly, the additional information may includea geographic location of the diagnosing physician or a patientassociated with the first diagnosis or a reading condition of thediagnosing physician associated with the first diagnosis, such as aworkstation configuration, a number of diagnosing physicians, or aresolution of the workstation.

The learning engine 110 may use a determined correlation in variousways. For example, in one embodiment, the learning engine 110 maygenerate a report based on one or more determined correlations (e.g., toreport on identified biases for one or more diagnosing physicians andoptionally one or more recommendations for addressing the biases).Alternatively or in addition, when a new image is received for analysis,the learning engine 110 may process the new image to determine whether adetermined correlation exists for the new image and, if so,automatically display a warning to a diagnosing physician regarding thecorrelation (e.g., while displaying the second image within an imagereview application). For example, when the learning engine 110determines a correlation between a bias of a diagnosing physician andthe prior five diagnoses generated by the diagnosing physician, thelearning engine 110 may display a warning why the prior five diagnosesof the diagnosing physician satisfy the correlation to help prevent thediagnosing physician from allowing previous diagnoses to impact acurrent diagnosis.

Similarly, when a new image is received for analysis and the correlationexists for the new image, the learning engine 110 may graphically markthe new image to indicate a portion of the new image that maypotentially lead to a discrepancy based on the correlation. For example,when the learning engine 110 determines a correlation between a bias ofa diagnosing physician and a particular abnormality within a particularregion of a lung, the learning engine 110 may add a graphical mark tonew images of the lung to mark the biased region of the lung. In someembodiments, the learning engine 110 may vary one or morecharacteristics of a graphical mark (e.g., color, size, shape,animation, etc.) to indicate a type of the correlation associated withthe graphical mark (e.g., a patient correlation, a time correlation, aprior diagnosis correlation, or an anatomical correlation). Also, insome embodiments, the learning engine 110 may vary one or morecharacteristics of a graphical mark based one or more configurablepreferences (e.g., of a user). In particular, in some embodiments, thepreferences may allow a user to enable or disable the graphical marks.Also, in some embodiments, when a user selects a graphical mark, thelearning engine 110 displays additional information regarding thecorrelation, such as a warning as described above.

In some embodiments, the learning engine 110 may also be configured toautomatically monitor the results of various imaging protocols andassociated clinical behavior. For example, the learning engine 110 maybe configured to automatically identify that in a body MRI protocol or aparticular series of images is never viewed by a physician or isroutinely diagnosed as being “normal.” Based on this information, thelearning engine 110 may recommend or automatically make changes to aprocedure to eliminate a particular series from the procedure, whichreduces radiation and data processing requirements. Similarly, when aparticular series in an imaging procedure is routinely diagnosed asbeing “abnormal,” the learning engine 110 may prioritize this series(e.g., for automatic processing, for manual review, or both).

For example, FIG. 13 illustrates a method 1300 performed by the sever102 (i.e., the electronic processor 104 executing instructions, such asthe learning engine 110) for automatically determining imagecharacteristics serving as a basis for a diagnosis associated with animage study type according to some embodiments. As illustrated in FIG.13, the method 1300 includes receiving, with the learning engine, animage study from at least one data source 112 over an interface (e.g.,the input/output interface 108) of the image study type (i.e., is aparticular type of image study) (at block 1302). The image studyincludes a plurality of images. As illustrated in FIG. 13, the method1300 also includes the determining, with the learning engine 110, animage characteristic for each of the plurality of images included in thereceived image study (at block 1304). The image characteristic mayinclude, for example, an anatomical structure, whether a contrast agentwas used or a radioactive isotope was used, a time after a contrastagent or a radioactive isotope was introduced, an orientation, an imageacquisition parameter, DICOM header information, or a combinationthereof.

The method 1300 also includes determining, with the learning engine 110,whether each of the plurality of images was used to establish adiagnosis (at block 1306). The learning engine 110 may determine whetheran image was used to establish a diagnosis by determining whether theimage includes an annotation, such as a graphical marker as describedabove or other diagnostic information. Alternatively or in addition, thelearning engine 110 may determine whether an image was used to establisha diagnosis by determining whether the image was displayed to adiagnosing physician rendering the diagnosis during a reading session ofthe diagnosing physician. Further yet, the learning engine 110 maydetermine whether an image was used to establish a diagnosis bydetermining whether the image was diagnosed as normal.

As illustrated in FIG. 13, the method 1300 also includes storing, withthe learning engine 110, the image characteristic for each of theplurality of images and an indicator of whether each of the plurality ofimages was used to establish the diagnosis in a data structure (at block1308). The learning engine 110 may use the generated data structure invarious ways. For example, the learning engine 110 may automaticallydetermine a frequency of an image with a particular image characteristicbeing included in a particular type of image study (e.g., this type ofimage is only included in this type of image study 50% of the time).Similarly, the learning engine 110 may automatically determine afrequency of an image with a particular image characteristic being usedto establish a diagnosis. For example, the learning engine 110 maydetermine a frequency of an image with a particular image characteristicincluding an annotation or not including an annotation, being displayedto a diagnosing physician or not being displayed to a diagnosingphysician, or being diagnosed as normal or not normal. The learningengine 110 may use these frequencies to generate a report. For example,the learning engine 110 may generate a report that includes a pluralityof frequencies associated with different image characteristics, such asfrequencies satisfying one or more configurable thresholds (e.g., imagecharacteristics being clinically-relevant at least 80% of the time orless than 50% of the time). The plurality of frequencies may beassociated with the same image study type or multiple image study types.Also, in some embodiments, the report may be parsable, such as by animaging modality, a diagnosing physician, an anatomical structure,patient demographic information, or a date.

The learning engine 110 may also use the data structure to modify asubsequent image study of the image study type. For example, when imageswith particular image characteristics are rarely or never reviewed, thelearning engine 110 may automatically eliminate those images from animage study (e.g., modify imaging parameters or discard images includedin a generated image study). The learning engine 110 may also use thedata structure to process subsequent image studies of the image studytype. For example, when a new image study of the image study type isreceived, the learning engine 110 may prioritize images included in thestudy based on the data structure. In particular, the learning engine110 may display images routinely serving as a basis for a diagnosisbefore images rarely serving as a basis for a diagnosis to improveefficiency and accuracy. Similarly, depending on what images areincluded in a received image study, the learning engine 110 may use thedata structure to prioritize image studies for review.

The functionality described above with respect to the learning engine110 has applicability in many areas of healthcare. For example, thelearning engine 110 may be used in orthopedic medicine. In particular,the learning engine 110 may be configured to automate implant planning,which provides time savings for both orthopedic surgeons and implantmanufacturers involved in the implant planning process. In particular,orthopedic practices are typically driven by their bottom lineefficiency and the number of surgeries that may be performed per week.Therefore, automating portions of this practice may improve efficienciesin orthopedic practices.

For example, FIG. 14 illustrates a method 1400 performed by the server102 (i.e., the electronic processor 104 executing instructions, such asthe learning engine 110) for automatically selecting an implant for apatient planning to undergo a procedure involving placement of theimplant according to some embodiments. As illustrated in FIG. 14, themethod 1400 includes receiving, with the learning engine 110, at leastone image of the patient from at least one data source 112 over aninterface (e.g., the input/output interface 108) (at block 1402). Forexample, a physician may order images of a body part of the patientreceiving an implant (e.g., using an electronic ordering system). Basedon the order, the patient undergoes an imaging study where medicalimages are acquired (e.g., radiographic images). When the imaging studyis complete, the captured images may be automatically sent to thelearning engine 110 for processing (e.g., when the images are stored,such as in a RIS).

As illustrated in FIG. 14, the method 1400 also includes receiving, withthe learning engine 110, an intended location of the implant withreference to the at least one image (at block 1404). In someembodiments, the intended location includes at least one anatomicalstructure or landmark, which may be manually specified by a user (e.g.,through one or more peripheral devices) or automatically identified bythe learning engine 110 as described above. When the learning engine 110automatically identifies the landmark, the learning engine 110 maydisplay the landmark to a user for approval or modification (e.g.,through one or more peripheral devices). As described above, thelearning engine 110 may use user modifications to anautomatically-identified landmark as feedback to improve future landmarkidentification.

As illustrated in FIG. 14, the method 1400 also includes automaticallydetermining, with the learning engine 110, an anatomical structure basedon the at least one image (at block 1406). For example, as describedabove with respect to FIG. 5, the learning engine 110 may be configuredto generate a diagnosis that includes an identification of an anatomicalstructure represented in an image. The method 1400 also includesdetermining, with the learning engine 110, a preference (at block 1408).The preference may include a preference of a physician performing theprocedure, a preference for a particular manufacturer of implants or aparticular supplier of implants, a preference for the procedure, anavailable implant, an available supplier or an available manufacturer, apreference of an organization, a preference of the patient, or acombination thereof. In some embodiments, as described above,preferences are configurable (e.g., by a user through a graphical userinterface).

The learning engine 110 then automatically selects one or more suggestedimplants based on the intended location, the anatomical structure, andthe preference (at block 1410). The learning engine 110 may display theone or more suggested implants through a graphical user interface (atblock 1412).

In some embodiments, the learning engine 110 may also be configured toautomatically add a suggested implant to the image. For example, thelearning engine 110 may be configured to generate a modified image thatincludes the anatomical structure and one of the suggested implantswhere the learning engine 110 automatically configures (e.g., adjusts ashape and size) and positions the implant based on the intended locationand is positioned based on the anatomical structure and the intendedlocation.

The learning engine 110 may display the modified image to a user, suchas within a graphical user interface. In some embodiments, a user mayadjust the selection, configuration, position, or a combination thereofof the implant included in the modified image (e.g., through one or moreperipheral devices). For example, the learning engine 110 may set astatus of the modified image to “preliminary plan,” and a physician(e.g., a surgeon) or an implant manufacturer or representative mayaccess the “preliminary plan” (e.g., by logging into a storage locationfor “preliminary plans” or by automatically receiving the “preliminaryplan,” such as within an e-mail message or other type of electronicnotification) to review the preliminary plan and make any desiredadjustments. When no adjustments are needed or after any desiredadjustments are made, the “preliminary plan” is approved. When a“preliminary plan” is approved, the learning engine 110 changes thestatus to “final” or “approved,” and, in some embodiments, the learningengine 110 automatically sends the approved implant details to amanufacturer for implant manufacturing (e.g., customization ifnecessary) and manufacturing or acquisition of associated implantguides. Accordingly, the learning engine 110 may be used in orthopedicmedicine to reduce turnaround time for receiving implants and humanerrors associated with such implant orders.

When adjustments are made to the “preliminary plan,” the learning engine110 may update any models used to select the implant or to determine theconfiguration and position of the implant. For example, in someembodiments, the learning engine 110 may develop a model based ontraining information as described above. The training information mayinclude a plurality of images associated with implants, and the learningengine 110 may perform machine learning to detect patterns betweenparticular implants and particular anatomical structures or otherimplant conditions (e.g., automatically learn how to identify anatomicallandmarks and place implants based on example images of anatomicallandmarks and associated implants positioned manually or automatically).Accordingly, the learning engine 110 may develop a model that canautomatically select suggested implants as described above. In someembodiments, the learning engine 110 may also develop models fordetermining an intended location, determining an anatomical structure,determining a preference, or a combination thereof. Also, in someembodiments, the learning engine 110 may update these models based onpost-procedure data (e.g., images, laboratory reports, a patientquestionnaire answers, or a follow-up appointment) associated with atleast one of the one or more suggested implants. Also, in someembodiments, the learning engine 110 may performance analytics orgenerate analytics or statistics associated with particular implants orimplant conditions, such as a usage rate of an implant, a success rateof an implant, most commonly-used implants, implants associated with thefewest patient complaints, most properly-placed implants, an implantmost commonly used for a particular type of patient, and the like.

Similarly, the learning engine 110 may be used to track clinicaloutcomes. Clinics, such as cancer centers often need to report onclinical outcomes. However, it is often difficult to map pathologyreports to image diagnoses. For example, a good pathology result mayrely on an accurate biopsy. However, biopsy accuracy may be limited byimage quality and equipment used to perform the biopsy procedure.Accordingly, the learning engine 110 may be configured to automaticallytrack clinical outcomes, which saves costs and may address accreditationneeds. For example, FIG. 15 illustrates a method 1500 performed by theserver 102 (i.e., the electronic processor 104 executing instructions,such as the learning engine 110) for mapping one or more biopsylocations to pathology results according to some embodiments.

As illustrated in FIG. 15, the method 1500 includes receiving, with thelearning engine 110, an image from at least one data source 115 over aninterface (e.g., the input/output interface 108) (at block 1502). Themethod 1500 also includes receiving, with the learning engine 110, abiopsy location (at block 1504). The biopsy location includes athree-dimensional position mapped to a position within the image. Insome embodiments, the learning engine 110 may automatically map a biopsylocation to a position with the image based on medical informationassociated with the patient (such as additional images, patientinformation, and procedure information). For example, as part of adiagnosis associated with the image, a lesion or other area of interestmay be graphically marked on the image, and the learning engine 110 mayuse this graphical reporting as described above to identify biopsylocations that may be specified for particular lesions or particularsegments of a lesion (e.g., top, right corner of the lesion). Similarly,in some embodiments, the learning engine 110 develops a model asdescribed above that map biopsy locations to particular locations withinan associated image using training information that includes imagesincluded marked biopsy locations (e.g., manually marked locations).

As illustrated in FIG. 15, the learning engine 110 automatically locatesan electronic pathology result for the biopsy location within the atleast one pathology result source over the interface (at block 1506).The pathology result source may include an EMR or a MS. In someembodiments, the learning engine 110 uses a model (e.g., developed usingmachine learning) to locate the electronic pathology result. The modelmay extract pathology location information from one or more pathologyreports and map each extracted pathology location to an associated imageand, optionally, a location within the associated image. Similarly, insome embodiments, the learning engine 110 locates the electronicpathology result by deducting semantics from text included in apathology report or searching for a field within a structured pathologyreport.

The learning engine 110 also generates an electronic correlation betweenthe biopsy location and the electronic pathology result (at block 1508).For example, in some embodiments, the learning engine 110 generates theelectronic correlation by generating a table or other data structurethat maps the biopsy location to the electronic pathology result. Asillustrated in FIG. 15, the learning engine 110 also displays the imagewith the biopsy location marked within the image (at block 1510). Forexample, FIG. 16 illustrates an image 1600 with two marked biopsylocations 1602. As illustrated in FIG. 16, the image 1600 provides auser, such as a radiologist with an image (e.g., a three-dimensionalimage) of an organ with associated biopsied locations. In someembodiments, when a user selects the marked biopsy location 1602, thelearning engine 110 automatically displays the corresponding electronicpathology result based on the electronic correlation. In someembodiments, the learning engine 110 displays the electronic pathologyresult within the associated pathology report (see, e.g., the examplepathology report 1700 illustrated in FIG. 17). Also, in someembodiments, the learning engine 110 displays other pathology reportswith the associated pathology report, such as past pathology reports,future pathology reports, or a combination thereof, to provide a userwith a timeline of pathology results. In some embodiments, a user mayenable or disable the marked biopsy locations 1602, such as usingconfigurable preferences associated with the user. Also, in someembodiments, a user can specify preferences for characteristics of themarked biopsy locations 1602 (e.g., size, shape, color, orientation,etc.).

By correlating images and pathology reports, the learning engine 110 canperform a comparison between an original diagnosis associated with animage (i.e., prior to the pathology results) and the resulting pathologyresult. As described above, the learning engine 110 may use thesecomparisons to identify common errors or biases of diagnosingphysicians. Similarly, the learning engine 110 may use these comparisonsto update models used to automatically generate diagnoses.

The learning engine 110 may also generate statistics and scores based onthese comparisons, such as a percentage of diagnoses consistent withcorresponding pathology results. For example, FIGS. 18 and 19 illustratestatistics that may be generated and displayed within graphical userinterfaces 1800 and 1900 in various formats (e.g., line graphs, bargraphs, and the like). The statistics and scores may be used tofacilitate internal quality assurance, such as by reviewing diagnosisand biopsy cases that are discordant and allowing a review of biopsylocations. Similarly, the statistics and scores may include a totalnumber of cases interpreted and the statistical results of such cases toprovide a quick snap-shot of the status of a clinical program. Also, insome embodiments, the learning engine 110 may generate a table thatcompares biopsy locations within images to corresponding electronicpathology results. For example, FIG. 20 illustrates a graphical userinterface 2000 that includes a table 2002 listing image studies andassociated pathology results.

As described above, the learning engine 110 may be configured to markbiopsy locations 1602 within an image, wherein the locations are linkedto the corresponding pathology results. Conversely, in some embodiments,a pathology report is interactive, such that a user can view pathologylocations within an image from the pathology report. For example, FIG.21 illustrates a method 2100 performed by the server 102 (i.e., theelectronic processor 104 executing instructions, such as the learningengine 110) for mapping pathology results to clinical images accordingto some embodiments.

Similar to the method 1500, the method 2100 includes receiving, with thelearning engine 110, an image from at least one data source 112 over aninterface (e.g., the input/output interface 108) (at block 2102),receiving, with the learning engine 110, a biopsy location includes athree-dimensional position mapped to a position within the image (atblock 2104), automatically locating, with the learning engine 110, anelectronic pathology result for the biopsy location within the at leastone pathology result source over an interface (at block 2106), andgenerating, with the learning engine 110, an electronic correlationbetween the biopsy location and the electronic pathology result (atblock 2108).

As illustrated in FIG. 21, the learning engine 110 displays anelectronic pathology report including the electronic pathology result(at block 2110), receives a selection of the electronic pathology resultfrom a user (at block 2112), and, in response to the selection,automatically, displays the image, wherein the biopsy location is markedwithin the image based on the electronic correlation (at block 2114).Accordingly, while viewing a pathology report, a user may be able toselect a particular pathology result (e.g., the result itself or anassociated link or icon) to review the corresponding biopsy locationwithin an image. In some embodiments, the learning engine 110 performsboth of the methods 1500 and 2100 to allow a user to toggle between andthe electronic pathology result and the associated marked biopsylocations within the associated image to navigate through biopsiedlocations.

Thus, the learning engine 110 may be configured to track and linkpathology results to exact biopsy locations within an image. Thistracking helps clinicians record, visualize, and locate pathologiesobtained using biopsy devices within an image. Accordingly, thistracking helps diagnosis, treat, and follow-up on disease progressionthrough a correlation between laboratory pathology results and medicalimaging of the same patient.

Thus, embodiments described herein provide methods and systems fordeveloping image analytics using machine learning for the healthcareindustry. The methods described herein are described as being performedby the server 102 and, in particular, by the learning 110 as executed bythe electronic processor 104. However, in some embodiments, one or moreof the methods, or portions thereof, may be performed by softwareexecuted by the electronic processor 104 distinct from the learningengine 110, a device separate from the server 102, or a combinationthereof. For example, in some embodiments, the learning engine 110 maydevelop the models as described above and a separate piece of softwareor a separate device may access the models to analyze images or performother functions.

Various features and advantages of the invention are set forth in thefollowing claims.

What is claimed is:
 1. A system for performing image analytics usinggraphical reporting associated with clinical images, the systemcomprising: at least one data source; and a server including anelectronic processor and an interface for communicating with the datasource the electronic processor configured to receive traininginformation from the at least one data source over the interface, thetraining information including a plurality of images and graphicalreporting associated with each of the plurality of images, eachgraphical reporting including a graphical marker designating a portionof one of the plurality of images and diagnostic information associatedwith the portion of the one of the plurality of images, and performmachine learning to develop a model using the training information,wherein the model is used to automatically analyze an image.
 2. Thesystem of claim 1, wherein the graphical marker includes aone-dimensional graphical marker, a two-dimensional graphical marker, ora three-dimensional graphical marker.
 3. The system of claim 1, whereinthe graphical marker was manually generated by a diagnosing physician.4. The system of claim 1, wherein the graphical marker was automaticallygenerated by a computer system.
 5. The system of claim 1, wherein thediagnostic information includes a classification of a plurality ofcategories.
 6. The system of claim 1, wherein the diagnostic informationincludes a normal category, an abnormal category, and an indeterminatecategory.
 7. The system of claim 1, wherein the diagnostic informationincludes a probability.
 8. The system of claim 1, wherein the diagnosticinformation includes a diagnosis.
 9. The system of claim 1, wherein thediagnostic information includes an anatomical structure.
 10. The systemof claim 1, wherein the diagnostic information includes a measurement ora location.
 11. The system of claim 1, wherein the diagnosticinformation includes morphological characteristics of an anomaly withinthe portion.
 12. The system of claim 1, wherein the training informationfurther includes a Digital Imaging and Communications in Medicine(DICOM) header file associated with at least one of the plurality ofimages.
 13. The system of claim 1, wherein the training informationfurther includes procedure information associated with at least one ofthe plurality of images.
 14. The system of claim 1, wherein the traininginformation further includes patient information associated with atleast one of the plurality of images.
 15. The system of claim 14,wherein the electronic processor is configured to receive the patientinformation from a hospital information system (HIS) or an electronicmedical record (EMR).
 16. The system of claim 1, wherein the traininginformation further includes a report associated with at least one ofthe plurality of images.
 17. The system of claim 16, wherein the reportincludes an order, a DICOM structured radiology report, a pathologyreport, or a test result.
 18. The system of claim 1, wherein thetraining information further includes an imaging property of a firstimage included in the plurality of images, the imaging propertyincluding at least one selected from a group consisting of a firstindication of whether a contrast agent was used, a second indication ofwhether a radioactive isotope was used, a time after an agent or isotopewas introduced, an orientation, and an image acquisition parameter. 19.The system of claim 1, wherein the at least one data source includes apicture archiving and communications system (PACS), a vendor neutralarchive, a radiology information system (RIS), an electronic medicalrecord (EMR), a hospital information system (HIS), an image studyordering system, or a computer assisted detection (CAD) system.
 20. Thesystem of claim 1, wherein the electronic processor is configured toperform the machine learning to develop the model using the traininginformation by evaluating a first portion designated by a firstgraphical marker within a first image included in the plurality ofimages on a second image included in the plurality of images, the secondimage acquired under a different imaging condition than the first image.21. The system of claim 20, wherein the different imaging conditionincludes at least one selected from a group consisting of a differentimage within the same exam, a different exam, a different time afterinjection of an agent or isotope, a different magnetic resonance imaging(MRI) acquisition parameter, and a different radiation parameter. 22.The system of claim 1, wherein the electronic processor is configured toperform the machine learning to develop the model using the traininginformation by comparing an anatomical position associated with a firstimage included in the plurality of images with an anatomical position ina second plurality of images acquired using different image acquisitionproperties than the first image.
 23. The system of claim 1, wherein theelectronic processor is configured to perform the machine learning todevelop the model using the training information by obtaining a clinicalresult associated with a first image included in the plurality ofimages, performing a comparison of the clinical result and thediagnostic information associated with the first image, assigning aweight to the diagnostic information based on the comparison, anddeveloping the model using the training information based on the weight.24. The system of claim 23, wherein the clinical result includes atleast one image.
 25. The system of claim 23, wherein the clinical resultincludes at least one laboratory result.
 26. The system of claim 1,further comprising updating the model based on a clinical resultassociated with a first image included in the plurality of images. 27.The system of claim 26, wherein the clinical result includes at leastone image.
 28. The system of claim 26, wherein the clinical resultinclude a laboratory result.
 29. The system of claim 1, furthercomprising updating the model based on feedback designating acorrectness of the training information.
 30. The system of claim 1,wherein the training information includes a data record specifyingcoordinates of a first graphical marker of a first image included in theplurality of images.
 31. The system of claim 1, wherein the traininginformation includes a first image included in the plurality of images,wherein a first graphical marker is superimposed on the first image andwherein the electronic processor is configured to automatically detectthe first graphical marker within the first image.